I had noticed before that the passwords generated by pwgen seemed to contain patterns: repeating vowels is something that stands out. I hadn't thought about it much, until I saw a colleague type pwgen -s at his terminal.

What is this -s?

As with most unix-y tools, a quick glance at the man page tells us everything we want to know. The -s is shorthand for --secure:

Generate completely random, hard-to-memorize passwords. These should only be used for machine passwords, since otherwise it's almost guaranteed that users will simply write the password on a piece of paper taped to the monitor...

As it turns out, pwgen actually doesn't generate secure passwords by default. The passwords it generates are designed to be easily memorized. That explains the patterns I noticed before!²

You can use it to define a $foo variable with the contents of bash variable $FOO, for example:

$ export FOO="bar"
$ jq -n --arg foo "$FOO" '{foo: $foo}'
{
  "foo": "bar"
}

You might notice the -n parameter: This tells jq to use null as input instead of reading json from stdin like it normally would. I'm using it in all examples on this page as it makes them a bit shorter. In real world usage, you probably won't need it.

As usual in bash scripts, you can also execute a command inline:

$ jq -n --arg hostname "$(hostname)" '{foo: $hostname}'
{
  "foo": "cypher"
}

Every variable created with --arg is treated as a string. That might cause some unexpected behavior when you work with numbers, like when you attempt to add a number to an argument. Comparisons between strings and numbers seem to work fine, but I wouldn't count on it (pun intended).

$ echo '{"a": 3.0}' | jq --arg foo 5 '$foo + .a'
jq: error (at <stdin>:1): string ("5") and number (3) cannot be added

There are two ways to solve this: either convert the variable to a number with tonumber or use --argjson instead. I prefer to use tonumber because that way I'm sure jq forces it to be a number. As my argument may be the result of an earlier command, I may be using an error message as input instead of the expected numeric result.

$ jq -n --arg foo 5 '($foo|tonumber) + 1'
6

$ jq -n --argjson foo 5 '$foo + 1'
6

Variables from files

It's also possible to load variables from files. With --slurpfile we can read a file containing json objects. An array containing the separate objects is then made available as a predefined variable. Note that even if your file contains only a single json object, as the variable is still an array, the [0] is required to access it.

$ cat foo.json
{"foo": "bar"}
{"bar": "foo"}

$ jq -n --slurpfile foo foo.json '$foo[0].foo'
"bar"

A second option is --rawfile, creating a string variable with the exact contents of the file. Note that as jq outputs json, reading and printing the file from the previous example results an escaped string.

$ jq -n --rawfile foo foo.json '$foo'
"{\"foo\": \"bar\"}\n{\"bar\": \"foo\"}\n"

That's it!

Converting using OpenSSL

The openssl command makes it easy to convert PFX files. It may ask you for the password if the private key was encrypted.

Export only the private key to a file named key.pem:

openssl pkcs12 -in cert.pfx -nocerts -out key.pem -nodes

Export only the public certificates to a file named cert.pem:

openssl pkcs12 -in cert.pfx -nokeys -out cert.pem

Export both public and private keys to bundle.pem:

openssl pkcs12 -in cert.pfx -out bundle.pem -nodes

• ssh yoursitebe@ssh.yoursite.be
• ssh yoursitebe@ssh123.webhosting.be

The first one may not work if your DNS records are not correct, the second one always works but is hard to remember. With just a small change to our local SSH config, we can use a third option: simply connect to your domain name!

• ssh yoursite.be

For this to work we have to tell ssh what to do with yoursite.be.

The local SSH config file

On computers running Linux and macOS, the configuration file can be found at:

~/.ssh/config

The file (or the .ssh directory) may not exist yet, you can create it manually.

Inside the file you can specify Host blocks to tell the SSH client what to do with certain hostnames. We'll use our domain name as that's the natural address to connect to when we want to update our website.

Host yoursite.be
	HostName ssh123.webhosting.be
	User yoursitebe

Host othersite.com
	HostName ssh472.webhosting.be
	User othersitecom

The correct values for the HostName and User parameters can be found after logging in to the My Combell control panel. Navigate to the correct web hosting account and click the SSH menu option on the left. You'll see the SSH host and SSH user name listed there.

After saving the file, you can connect right away:

ssh yoursite.be
Last login: Mon Dec 16 14:28:14 2019 from 178-221-32-24.access.telenet.be
yoursitebe@com-linssh123:~#

This also works when using scp or rsync to transfer files!

As I'm using Cloudflare to cache my website my first guess was that the cache on their side wasn't being cleared. Pushed the clear cache button multiple times without any effect. It's clearly broken! A direct request to my webserver cleared any blame on Cloudflare though: my server was replying with an old version of my CSS file.

Then it must be rsync! I'm using rsync to push the new versions of my website from my development machine to the server. A few retries later the funny looking webpages suggested the CSS was still wrong. Checking the file on my server confirmed that rsync had actually replaced the CSS, retrying the upload was useless.

Why was nginx still serving the old CSS? Turns out it was my configuration.

Sendfile

As recommended by most optimization guides my nginx configuration allows using sendfile. This option is especially useful for static websites like mine, as every single page can be served right from the filesystem.

http {
  sendfile on;
  tcp_nopush on;
}

But what does it do exactly?

The nginx documentation explains the sendfile option as:

Enables or disables the use of sendfile().

That's exactly what I would expect from a boolean option! Sendfile is actually a Linux system call executed by nginx, for more information we have to resort to the man pages.

sendfile() copies data between one file descriptor and another. Because this copying is done within the kernel, sendfile() is more efficient than the combination of read(2) and write(2), which would require transferring data to and from user space.

So basically nginx just asks the kernel to copy the contents of a file to the connection with the visitor's browser. This makes serving static files a lot faster, as it eliminates the step where nginx copies the file into its own buffers. Yay!

Open file cache

Another optimization I've been using is the open_file_cache options:

http {
  open_file_cache max=1000 inactive=20s;
  open_file_cache_errors on;
  open_file_cache_min_uses 2;
  open_file_cache_valid 30s;
}

When we turn to the nginx documentation again, what we should know about the open file cache is that it contains information about files being served from the filesystem:

• open file descriptors, their sizes and modification times
• information on existence of directories
• file lookup errors

This cache is quite useful as the modification time allows nginx to reply to If-Modified-Since requests without actually opening the file. Another situation that happens quite frequently is a non-existing file being requested multiple times. The best known example is the favicon.ico that most browsers request whenever you visit a website. As nginx already knows that the file is not actually there, it can simply reply with a 404 response right away.

Great, huh?

Remember how sendfile copies data from one file descriptor to another? If the file on disk was replaced (not just modified) then this causes nginx to keep sending data from the old file instead of the new one. This can be very confusing if you also have a web cache (Varnish, or a service like Cloudflare for example) as you'll be blaming the cache first.

Looks like there is no way to clear the open file cache. As I don't want to restart nginx every time I update my website I have modified my configuration to no longer cache files for 30s.

Rsync

As an extra: rsync is the reason why my files were replaced.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

This means the existing file is not modified. It is actually replaced by a new file, to make sure we don't end up with only half of the contents in case transfer fails.

Hello Docker!

Then Docker came and brought a new way to manage services. With the Dockerfile it introduced a standardized way to describe exactly what should be running inside a container. While containers were nothing new, the easy to use of the Docker toolchain changed everything.

If you're just managing a few servers and are not planning to scale using Kubernetes or similar orchestration tools, using docker can make managing your services harder. Keeping containers up to date means you have to manage another package manager and configuration depends on whatever the image maintainer decided to add to the image. Having a bit more flexibility in the software you're running is awesome, though! That's why I decided to combine services running inside Docker containers with traditional configuration management.

Hybrid configuration

Since I want to use my existing Ansible code to manage nginx and its configuration, I need to be able to:

• Start, stop and reload my nginx service using systemd
• Configure nginx in /etc/nginx
• View and cycle logs in /var/log/nginx
• Connect to local services outside the container (php-fpm, Go webservices, ..)
• Publish websites with document roots in /var/www

But I also want features that I can't get on my standard debian system like TLS 1.3 and brotli compression.

Managing a containerized nginx service with systemd

First step was to create an nginx docker image with support for everything I want. The systemd unit below takes care of downloading and starting the container on my server. Volumes are mounted on every important path listed above to make sure I can configure everything as if the nginx service was installed using the default Debian package.

[Unit]
Description=Nginx
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
Restart=always
ExecStartPre=/usr/bin/docker pull sorcix/nginx:latest
ExecStart=/usr/bin/docker run --net=host --name nginx --rm -v /etc/nginx:/etc/nginx:ro -v /var/log/nginx:/var/log/nginx:rw -v /etc/ssl:/etc/ssl:ro -v /var/www:/var/www sorcix/nginx:latest
ExecReload=/usr/bin/docker exec nginx nginx -s reload
ExecStop=/usr/bin/docker exec nginx nginx -s quit

[Install]
WantedBy=multi-user.target

Networking & security

The --net=host part makes the container use host networking instead of the separate docker network that is usually created for containers. Forwarding ports 80 and 443 is not required (which actually makes traffic handling a little faster) and we can connect to services running on the host. Note that this also means that the additional security that containers usually provide is now missing. That's fine with me as I'm using this as a drop in replacement for the default Debian nginx package which does not provide containerization at all.

Updates

The ExecStartPre command downloads new versions of the sorcix/nginx container every time I restart nginx. As I manage the container image myself this isn't really an issue, but you should disable this when using images created by others.

ssh -J charmeleon.mydomain.net bulbasaur.mydomain.net

As we're connecting to Bulbasaur a lot, we may want to tell OpenSSH that we're using charmeleon as a jumphost for bulbasaur:

# ~/.ssh/config

Host bulbasaur.mydomain.net
	ProxyJump charmeleon.mydomain.net

With this configuration in place, we can simply use ssh bulbasaur.mydomain.net. This also works for scp, rsync and everything else that uses your local SSH client.

Working with legacy

On versions of OpenSSH before 7.3 the ProxyJump option is not available. In that case you'll have to use an alternative configuration using the ProxyCommand option to specify the exact command OpenSSH has to execute to set up the tunnel.

Host bulbasaur.mydomain.net
	ProxyCommand ssh charmeleon.mydomain.net -W %h:%p

Oops: wildcards

Watch out when you're using wildcards in your ssh config:

Host *.mydomain.net
	ProxyJump charmeleon.mydomain.net

Using ssh server.mydomain.net now causes a loop as the connection to the jumphost also matches the block and tries to use the jumphost.

An easy way to solve this is an alias for the jumphost:

Host charmeleon
	HostName charmeleon.mydomain.net

Host *.mydomain.net
	ProxyJump charmeleon

• Restic is very simple to use. It's a cli-tool with a few easy to understand commands and has good documentation including examples.
• You don't need a special backup service. Restic runs on the host being backed up, and is happy to store its backups on a simple file server or cloud service.
• Security is important and the developers have written a decent threat model: trust only the host where restic runs, which makes sense as it contains the data anyway. The place where you store your backups is always considered insecure.

Mounting storage space

TransIP is giving 1TB of free storage on their Stack storage service, which is more than enough for my backup needs. The downside is that the only useful way to access it is a painfully slow webdav mount. If you're backing up data that changes fast and need a consistent state, I would suggest choosing a different place to store your data. For most of my servers I don't really care about backup speeds, though.

Mounting the volume using an entry in /etc/fstab would work, but as I want to specify dependencies later I opt to create a Mount Unit instead. This is basically the systemd version of a single fstab entry.

# /etc/systemd/systemd/mnt-backups.mount
[Unit]
Description=Backup storage
After=network-online.target
Wants=network-online.target

[Mount]
Type=davfs
What=https://something.stackstorage.com/remote.php/webdav/
Where=/mnt/backups
Options=uid=root,gid=root,file_mode=644,dir_mode=755,nosuid,noexec,rw
TiemoutSec=15

[Install]
WantedBy=multi-user.target

If you're using webdav (using davfs2) like me, you'll also have to set your username and password in the secrets file /etc/davfs2/secrets. Be sure to restrict the permissions on this file as it contains passwords.

https://something.stackstorage.com/remote.php/webdav/ username password

We can now tell systemd to mount our volume:

systemctl start mnt-backups.mount

And also set it to be mounted every time our machine boots:

systemctl enable mnt-backups.mount

Initialize the repository

Before our first backup we have to initialize a repository. This creates the directory structure and configuration that restic will use to store our backups. Note that, as restic does not trust our backup volume, any configuration stored there is also encrypted!

restic init --repo /mnt/backups/myserver

Do not forget the password you enter during this step. Every single file on the backup volume is encrypted and without the password your backups become useless.

Backup service

The Service Unit shown below describes a service that creates a single backup and stops.

# /etc/systemd/system/backup.service
[Unit]
Description=Backup
After=mnt-backups.mount
Wants=mnt-backups.mount

[Service]
Type=simple
Environment=RESTIC_PASSWORD=mysupers3cr3tpassw0rd!
Environment=RESTIC_REPOSITORY=/mnt/backups/myserver
ExecStart=/usr/local/bin/restic backup /home /srv/web

In the [Unit] section we specify that the mount we created earlier is a requirement for making backups (Wants). The After setting is used to order the units when both are set to start when booting the system. It's a bit redundant to specify that the Service should be after the Mount as we already specified a dependency, but it's a common thing to do.

Note that I only backup my /home and /srv/web folders as everything else can either be recreated or is managed by Ansible on my servers. If you prefer to have a copy of your entire filesystem remember to change the path!

We can now create a new backup manually by running systemctl start backup. This isn't really useful by itself as we could have run restic backup manually just as easy, with the exception that we don't have to type the password anymore. Once we combine this with a systemd timer, things become interesting!

Remember that we set the backup volume to be mounted on boot? That isn't really required. As we have specified a dependency on the volume, systemd will check if it's mounted before starting the backup service. The free Stack volume I'm using is a bit wonky though, so I prefer to keep it mounted as that's a bit more stable.

Creating backups automatically

Now that our backup works, it's time to have it create a backup automatically every night. Creating a Timer Unit with the same name as the service we want to start does exactly that!

# /etc/systemd/system/backup.timer
[Unit]
Description=Backup daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

This simple timer is set to run daily at midnight. Setting the Persistent attribute tells systemd that if our host was down at midnight, it has to run the backup as soon as it comes up again. As servers are usually online 24/7 this doesn't really matter but it is a great setting for desktops and laptops.

Enable the timer using systemctl:

systemctl enable backup.timer

Now enjoy having a fresh backup of your server every night! Similar to the backup service, you can also create a service to remove your old snapshots. By default Restic just keeps adding data and never removes anything, this may become a problem pretty fast if you're backup up a lot of data.

Restoring

As with any backup system: remember to test restoring your backups regularly. Restic gives us two great options in case we fuck up and need our data back:

• Restore a complete snapshot if you know to which point in time you need to return and if you need a copy of all your files.
• Mount your backup repository as a fuse filesystem, allowing you to browse all your snapshots and the files within as if they're all just there on your server.

Configuring a user unit

The directory structure that holds unit files is similar to the systemwide configuration, but is located in a user's home directory:

$ mkdir -p ~/.config/systemd/user

In the user folder we can create service units with a .service extension. A unit is a ini-style file describing a service that has to be managed.

Let's say we have a Laravel PHP queue worker that we want to be managed by systemd, called worker.service:

[Unit]
Description=Laravel PHP queue worker

[Service]
ExecStart=/usr/bin/env php /path/to/my/project/artisan queue:work --sleep=3 --tries=3

[Install]
WantedBy=default.target

Run $ systemctl --user daemon-reload to make systemd recognize your changes.

We can now manage this service with systemd:

• $ systemctl --user start worker starts the service
• $ systemctl --user stop worker stops the service
• $ systemctl --user restart worker restarts the workers
• $ systemctl --user status worker shows us the service status

If we want this service to start automatically when our user logs in, we have to enable it:

• $ systemctl --user enable worker
• $ systemctl --user disable worker

Viewing logs

Everything our PHP script writes to stdout and stderr will be recorded by journald. We can view the logs for this specific service:

$ journalctl -u worker

On older versions of systemd (like the one shipped with Debian Jessie), you might need to use a flag specific to user units:

$ journalctl --user-unit worker

Automatically restarting on failure

Let's be honest: things go wrong. When our worker crashes we want to make sure it's started again. Fortunately, we can configure systemd to restart on failure:

Add the following to the [Service] section of the unit file:

RestartSec=10s
Restart=on-failure

This makes systemd restart the service 10 seconds after it has stopped with a non-zero exit code. If the process exited cleanly, systemd considers the work as done and won't restart the service until you manually start it later.

Starting user services at boot

By default, systemd starts enabled user units when the user logs in, and stops them once the user has logged out. This is great for desktop pcs where a user wants to run things while he's using the computer. On servers, however, we may want to run those units all the time.

# loginctl enable-linger USERNAME

You can check the status for each user by:

# loginctl user-status USERNAME

It should show that the user is lingering:

vic (1000)
           Since: Fri 2017-09-01 21:54:29 CEST; 3s ago
           State: lingering
            Unit: user-1000.slice

When the user is not logged in and does not have lingering enabled, loginctl doesn't know about any user sessions so it will return an error:

Failed to get user: No user 1000 known or logged in

Makefile 101

In case you've worked with make before, skip this section! If not: let's see what a basic makefile looks like.

Make defines targets, these are things that need to be done. Usually a target represents a file that has to be created. Every target contains a list of commands that have to be executed, we call that the recipe. Make will only execute the recipe if a file with the same name as the target does not exist. If your target does not represent a file, it's a phony target.

mytarget:
	echo Hello world.

space: mytarget foo
	echo "I'm in space."

foo:
	echo I am run only once
	touch foo

.PHONY: mytarget space

The above makefile contains 3 targets: mytarget, foo and space. The second target tells us that it needs both other targets, so when you call make space, make will also execute mytarget and foo.

At the bottom of the file we define a special .PHONY target: Here we simply tell make which targets are phony targets. Those do not represent a file on disk. As we usually don't compile separate Go files, most of our targets will be phony targets.

Let's see what this does: save the file as Makefile and run make space in the same directory:

$ make space  
echo Hello world.
Hello world.
echo I am run only once
I am run only once
echo "I'm in space."
I'm in space.

Notice that make always tells us what command it is executing. You can hide a command by prefixing it with @. As we'll be executing multiple targets concurrently, it may be useful to see which commands are running.

Run make space again. You'll see that the foo target is not run anymore. That's because we created a file named foo and foo is not a phony target.

Variables

GNU Make has two types of variables. When we assign to a variable with := the contents is evaluated at the time the variable is declared. If we use just = the contents is evaluated every time we use the variable.

MYVAR := Hello there! I am currently in $(shell pwd)
PWD = $(shell pwd)

The pwd shell command will be executed every time we use variable $(PWD) in the makefile, while $(MYVAR) will always contain the same value.

Cross compiling

Make supports running multiple targets at the same time. This means that we need a separate target for every GOOS and GOARCH in order to take advantage of the concurrency.

A simple way to do this would be:

linux:
	GOOS=linux GOARCH=amd64 go build ...

windows:
	GOOS=windows GOARCH=amd64 go build ...

release: windows linux

.PHONY release windows linux

This builds both linux and windows binaries when we call make release. We can improve this Makefile by looping over the platforms and architectures we want to build:

PLATFORMS := linux/amd64 windows/amd64

release: $(PLATFORMS)

$(PLATFORMS):
	echo Hello, I am $@

.PHONY release $(PLATFORMS)

What we do here is create a target that contains all possible platforms. Every platform is a separate target, just like it was in the previous Makefile. That also means that we have to add every platform to the phony list. Luckily, we can do that using the same variable. Inside the recipe we can use the special $@ that contains the name of the current target.

The next part is a bit harder, we need to separate OS and ARCH. Remember that variables can be evaluated at runtime? We'll use that feature to create an $(os) and $(arch) var.

temp = $(subst /, ,$@)
os = $(word 1, $(temp))
arch = $(word 2, $(temp))

String manipulation functions in make are limited, so we have to be a bit creative here. We'll use these variables inside the $(PLATFORM) target. Remember that $@ contains the name of the current target? When we replace the slash in this string we can use the word function to get the first and last part.

Combined with our previous Makefile, this becomes:

PLATFORMS := linux/amd64 windows/amd64

temp = $(subst /, ,$@)
os = $(word 1, $(temp))
arch = $(word 2, $(temp))

release: $(PLATFORMS)

$(PLATFORMS):
	GOOS=$(os) GOARCH=$(arch) go build -o '$(os)-$(arch)' mypackage

.PHONY release $(PLATFORMS)

Note that variables can only be set outside the recipe. As the contents is only evaluated when we use them, it still contains the correct value.

Concurrency

We now have a Makefile that creates binaries for the platforms we listed in the $(PLATFORMS) variable. When we call make release it builds everything we asked, one by one. The main reason to use make is the easy concurrency: simply call make with the -j n argument, where n is the number of simultaneous jobs.