Recapping everything with an old final exercise
This week the goal was to practice everything that we have learned during the course by doing one of the final lab exercises from an older course implementation.
The task
I chose the task found here and it translate freely like this:
- Install a Linux workspace and prepare it for remote management.
- Setup a firewall.
- Create users for Joe Doe, Jorma Mähkylä, Pekka Hurme, Ronaldo Smith, Håkan Petersson and Einari Mikkonen.
- Create an example homepage for every user.
- Install LAMP - Linux, Apache, MySQL, PHP. Because we have not used PHP on this course I will instead be using Python Flask in it's place.
- Create an example program that shows something from the database.
- Set the example program visible on the url: http://invis.example.com.
- Create a new command that works on all users like all default commands do that prints the time.
- Create a metapackage that installs git, httpie, curl, mitmproxy.
- Create a static html page on the url: http://unikarhu.example.com.
- Use a tool to measure the load on the computer and then load test it and find the logs from the loead test.
- Change the name of the metapackage
Installation and setup
I installed a Debian 11.0.0-amd64-xfce from an iso file on a virtual machine on VirtualBox. I created the user juuso and set the domain name to example.com on installation.
Then I ran the command sudo apt-get update && sudo apt-get install to make sure that everything is up to date, but I got an error that my user is not in the sudoers file. To fix it I logged in as root with su root and added the user juuso to sudoers with sudo usermod -aG sudo juuso. Then I logged out and back in and checked my groups. Then I could run the update command again successfully.
Remote access
To setup remote access I installed ssh and ufw and set them up so that ssh connections are allowed and root login on ssh is not permitted. I can't really check the firewall from a virtual machine but I'll set it up either way.
sudo apt-get install -y ufw ssh
sudoedit /etc/ssh/sshd_config
sudo service ssh restart
sudo ufw allow 22/tcp
sudo ufw enable
Apache and Flask
Next I installed Apache and Python Flask with sudo apt-get install -y apache2 python3 python3-flask and checked that the default Apache landing page was visible.
I then added a rule to UFW that allows traffic to port 80, so that the server would be visible if it was accessible on other machines sudo ufw allow 80/tcp && sudo systemctl restart ufw.
Then I tested that Flask worked with a basic hello world app. I created a public_html folder for this with the hello.py inside as the same structure will be used for every user later on.
from flask import Flask
app = Flask(__name__)
@app.route("/")
def hello():
return "Hello World!"
app.run(debug=true)
Then I enabled the userdir mod for with sudo a2enmod userdir && sudo systemctl restart apache2 Apache so that I could get the example user pages working for every user. I also created an index.html file inside the public_html folder just to see that the mod works.
To serve Flask apps in user home pages I decided to use CGI to run the Python scripts that render the pages. I'm following this post on how to set up the page on a single user. And I'm following this post on how to enable CGI execution for Apache.
First I'll enable the mod on Apache with sudo a2enmod cgi and then allow running CGI-scripts in users' public_html directories by adding this to /etc/apache2/apache2.conf. I also had to enable the rewrite mod with sudo a2enmod rewrite because it is used in the .htaccess file.
<Directory "/home/*/public_html">
Options +ExecCGI
AddHandler cgi-script .cgi
</Directory>
Then I added the following files inside my public_html folder. The main.cgi file needs to be set with permissions to run with sudo chmod +x main.cgi
.htaccess
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ /home/USER/public_html/main.cgi/$1 [L]
main.cgi
#!/usr/bin/python3
import importlib.util
from wsgiref.handlers import CGIHandler
spec = importlib.util.spec_from_file_location("site", "/home/USER/public_html/site.py")
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
app = mod.app
CGIHandler().run(app)
site.py
from flask import Flask
app = Flask(__name__)
@app.route("/")
def hello():
return "Hello USER!"
if __name__ == "__main__":
app.run()
Now the hello Flask app is working from the public_html directory.
Users and the skeleton files
Before creating the users I modified the skeleton files in /etc/skel that will be used when creating the users' home directories so that there will be a public_html directory with an site.py file ready to see if user pages work on Apache when it is installed end set up. The example homepage will be the same as above. I will also create a script that replaces temporary strings with the user in the .htacces and main.cgi files.
sudo mkdir /etc/skel/public_html
sudoedit /etc/skel/public_html/.htaccess
sudoedit /etc/skel/public_html/main.cgi
sudoedit /etc/skel/public_html/site.py
sudo chmod +x /etc/skel/public_html/main.cgi
sudoedit /usr/local/sbin/adduser.local #this is run every time adduser is finished
#!/bin/bash
sed -i "s/USER/$1/" $4/public_html/.htaccess
sed -i "s/USER/$1/" $4/public_html/main.cgi
sed -i "s/USER/$1/" $4/public_html/site.py
Now I'll try creating a user for Einari and see that the skeleton files are used. I used pwgen to create the password.¨
pwgen 20 1
# example output: ePhe6ughuVieBiFiesha
sudo adduser einari
ls /home/einari/public_html
# main.cgi site.py
Then I created all the other users the same way. I checked that the file tree looked correct with the tree command. And the user pages were visible on Firefox!
MariaDB for Einari
To install MariaDB I ran sudo apt-get install -y mariadb-server and then ran the script for secure installation that came with MariaDB sudo mysql_secure_installation.
Then I created a test database as root in MariaDB by logging in with sudo mariadb. I also added a user for Einari that authenticaes either by the linux user or by a password. Then I added Einari all privileges on the test database.
MariaDB [(none)]> CREATE DATABASE test;
Query OK, 1 row affected (0.003 sec)
MariaDB [(none)]> USE test;
Database changed
MariaDB [test]> CREATE TABLE books ( BookID INT NOT NULL PRIMARY KEY AUTO_INCREMENT, Title VARCHAR(100) NOT NULL, Author VARCHAR(100) NOT NULL);
Query OK, 0 rows affected (0.011 sec)
MariaDB [test]> INSERT INTO books (Title, Author)
-> Values('The Fellowship of the Ring', 'J.R.R. Tolkien'),
-> ('Make: Sensors: A Hands-On Primer for Monitoring the Real World with Arduino and Rapberry Pi', 'Tero Karvinen');
Query OK, 2 rows affected (0.001 sec)
Records: 2 Duplicates: 0 Warnings: 0
MariaDB [test]> SELECT * FROM books;
+--------+---------------------------------------------------------------------------------------------+----------------+
| BookID | Title | Author |
+--------+---------------------------------------------------------------------------------------------+----------------+
| 1 | The Fellowship of the Ring | J.R.R. Tolkien |
| 2 | Make: Sensors: A Hands-On Primer for Monitoring the Real World with Arduino and Rapberry Pi | Tero Karvinen |
+--------+---------------------------------------------------------------------------------------------+----------------+
2 rows in set (0.000 sec)
MariaDB [(none)]> CREATE USER 'einari'@'localhost' IDENTIFIED VIA unix_socket OR mysql_native_password USING 'invalid';
Query OK, 0 rows affected (0.002 sec)
/*the real password generated with pwgen*/
MariaDB [(none)]> SET PASSWORD FOR 'einari'@'localhost' = PASSWORD('supersecretpassword');
Query OK, 0 rows affected (0.003 sec)
MariaDB [(none)]> GRANT ALL ON test.* TO einari@localhost;
Query OK, 0 rows affected (0.005 sec)
Then to test that Einari could in fact use the database I logged in temporarily as Einari with su einari;
WSGI app for Einari on invis.example.com
While I was still logged in as Einari I created a wsgi_app folder along with dbapp.wsgi and books.py inside it in his home directory.
dbapp.wsgi
import sys
assert sys.version_info.major >= 3, "Python version too old in dbapp.wsgi!"
sys.path.insert(0, '/home/einari/wsgi_app/')
from books import app as application
books.py
from flask import Flask
app = Flask(__name__)
@app.route("/")
def hello():
return "Hello wsgi!"
Then I set up WSGI the same as on my old blog post. I Installed the WSGI module and added a site config in the sites-available folder. In the /etc/hosts I added 127.0.0.1 invis.example.com so that requests to that url are redirected to localhost and the site is visible while testing.
sudo apt-get -y install libapache2-mod-wsgi-py3
sudoedit /etc/apache2/sites-available/invis.example.com.conf
sudo a2ensite invis.example.com.conf
sudo systemctl restart apache2
sudoedit /etc/hosts
<VirtualHost *:80>
ServerName invis.example.com
WSGIDaemonProcess einari user=einari group=einari threads=5
WSGIScriptAlias / /home/einari/wsgi_app/dbapp.wsgi
<Directory /home/einari/wsgi_app/>
WSGIScriptReloading On
WSGIProcessGroup einari
WSGIApplicationGroup %{GLOBAL}
Require all granted
</Directory>
</VirtualHost>
Now that the WSGI app worked I created a basic app that reads the test database I created earlier. I created in in part according to this post by Tero Karvinen.
books.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'mysql://einari:supersecretpassword@localhost/test'
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
db = SQLAlchemy(app)
class Books(db.Model):
BookId = db.Column(db.Integer, primary_key=True)
Title = db.Column(db.String(100), nullable=False)
Author = db.Column(db.String(100), nullable=False)
@app.before_first_request
def beforeFirstRequest():
db.create_all()
@app.route("/list")
def list():
books = Books.query.all()
str = ""
for book in books:
str += book.Author + ": " + book.Title + "<br>"
return str
@app.route("/hello")
def hello():
return "Hello wsgi!"
New command "telltime"
I created a new file in my home directory called telltime and created a basic script that echoes the current time. I set the file permissions so that everyone can read end execut it.
I then copied the script to /usr/bin with sudo cp telltime /user/bin and tested the script with einari user.
Meta package
I created the meta package according to this post by Tero Karvinen. First I installed the equivs package and created a skeleton file with it. I then edited the file so that the uncommented lines contained the following:
Section: misc
Priority: optional
Standards-Version: 3.9.2
Package: juusos-pkg
Version: 0.1
Depends: git, httpie, curl, mitmproxy
Description:
long description and info
.
second paragraph
To build it I ran equivs-build juusos-pkg.cfg and got the following files as output.
- juusos-pkg_0.1_all.deb
- juusos-pkg_0.1_amd64.buildinfo
- juusos-pkg_0.1_amd64.changes
I first had to install gdebi with the package manager and then tried running sudo gdebi -n juusos-pkg_0.1_all.deb. I think the install went succesfully because I got this message at the end.
Static site unikarhu.example.com
I added a new line to the hosts file the same as for invis.example.com and then added a new site config in the sites_available folder and then enabled it with sudo a2ensite.
unikarhu.example.com.conf
<VirtualHost *:80>
ServerName unikarhu.example.com
ServerAlias unikarhu.example.com
DocumentRoot /home/juuso/static_site
<Directory /home/juuso/static_site>
Require all granted
</Directory>
</VirtualHost>
/home/juuso/static_site/index.html
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Static html</title>
</head>
<body>
<p>Static html5 page</p>
</body>
</html>
Load testing
To load test I ran while true; do uptime; sleep 30; done on one terminal so I could see a running load average while running a simulated requests on invis.example.com on another terminal with ab -kc 100 -t 60 http://invis.example.com/. It keeps 100 concurrent connections and requests the url 50000 times with a timeout limit of 60 seconds.
while true; do uptime; sleep 30; done
14:25:19 up 2:17, 1 user, load average: 0.01, 0.06, 0.08
14:25:49 up 2:17, 1 user, load average: 0.01, 0.06, 0.08
14:26:19 up 2:18, 1 user, load average: 0.00, 0.05, 0.07
14:26:49 up 2:18, 1 user, load average: 0.07, 0.06, 0.08
14:27:19 up 2:19, 1 user, load average: 0.04, 0.05, 0.07
14:27:49 up 2:19, 1 user, load average: 0.10, 0.06, 0.08
14:28:19 up 2:20, 1 user, load average: 0.11, 0.07, 0.08
14:28:49 up 2:20, 1 user, load average: 1.95, 0.52, 0.23
14:29:19 up 2:21, 1 user, load average: 1.25, 0.48, 0.22
14:29:49 up 2:21, 1 user, load average: 0.76, 0.43, 0.21
14:30:19 up 2:22, 1 user, load average: 0.86, 0.47, 0.23
14:30:49 up 2:22, 1 user, load average: 2.42, 0.89, 0.38
14:31:19 up 2:23, 1 user, load average: 2.03, 0.96, 0.42
14:31:49 up 2:23, 1 user, load average: 1.29, 0.88, 0.41
It seems that the load levels were quite low even while serving 50000 requests within a little over 20 seconds. But it seems some of the requests failed, though just 22 in this last test.
Changing the metapackage name
To change the package name I changed the name on the Package-line in the juuso-pkg.cfg file to xoy-tools, the version to 0.2 and ran the equivs-build command again. It created these files:
- xoy-tools_0.2_all.deb
- xoy-tools_0.2_amd64.changes
- xoy-tools_0.2_amd64.buildinfo