Using Restic and systemd to backup a linux host

by on

Since moving away from Digital Ocean (and their snapshots) I’ve been looking for a good way to create backups for my servers. I’ve recently stumbled upon the Restic project and have been using it ever since.

Mounting storage space

TransIP is giving 1TB of free storage on their Stack storage service, which is more than enough for my backup needs. The downside is that the only useful way to access it is a painfully slow webdav mount. If you’re backing up data that changes fast and need a consistent state, I would suggest choosing a different place to store your data. For most of my servers I don’t really care about backup speeds, though.

Mounting the volume using an entry in /etc/fstab would work, but as I want to specify dependencies later I opt to create a Mount Unit instead. This is basically the systemd version of a single fstab entry.

# /etc/systemd/systemd/mnt-backups.mount
[Unit]
Description=Backup storage
After=network-online.target
Wants=network-online.target

[Mount]
Type=davfs
What=https://something.stackstorage.com/remote.php/webdav/
Where=/mnt/backups
Options=uid=root,gid=root,file_mode=644,dir_mode=755,nosuid,noexec,rw
TiemoutSec=15

[Install]
WantedBy=multi-user.target

If you’re using webdav (using davfs2) like me, you’ll also have to set your username and password in the secrets file /etc/davfs2/secrets. Be sure to restrict the permissions on this file as it contains passwords.

https://something.stackstorage.com/remote.php/webdav/ username password

We can now tell systemd to mount our volume:

systemctl start mnt-backups.mount

And also set it to be mounted every time our machine boots:

systemctl enable mnt-backups.mount

Initialize the repository

Before our first backup we have to initialize a repository. This creates the directory structure and configuration that restic will use to store our backups. Note that, as restic does not trust our backup volume, any configuration stored there is also encrypted!

restic init --repo /mnt/backups/myserver

Do not forget the password you enter during this step. Every single file on the backup volume is encrypted and without the password your backups become useless.

Backup service

The Service Unit shown below describes a service that creates a single backup and stops.

# /etc/systemd/system/backup.service
[Unit]
Description=Backup
After=mnt-backups.mount
Wants=mnt-backups.mount

[Service]
Type=simple
Environment=RESTIC_PASSWORD=mysupers3cr3tpassw0rd!
Environment=RESTIC_REPOSITORY=/mnt/backups/myserver
ExecStart=/usr/local/bin/restic backup /home /srv/web

In the [Unit] section we specify that the mount we created earlier is a requirement for making backups (Wants). The After setting is used to order the units when both are set to start when booting the system. It’s a bit redundant to specify that the Service should be after the Mount as we already specified a dependency, but it’s a common thing to do.

Note that I only backup my /home and /srv/web folders as everything else can either be recreated or is managed by Ansible on my servers. If you prefer to have a copy of your entire filesystem remember to change the path!

We can now create a new backup manually by running systemctl start backup. This isn’t really useful by itself as we could have run restic backup manually just as easy, with the exception that we don’t have to type the password anymore. Once we combine this with a systemd timer, things become interesting!

Remember that we set the backup volume to be mounted on boot? That isn’t really required. As we have specified a dependency on the volume, systemd will check if it’s mounted before starting the backup service. The free Stack volume I’m using is a bit wonky though, so I prefer to keep it mounted as that’s a bit more stable.

Creating backups automatically

Now that our backup works, it’s time to have it create a backup automatically every night. Creating a Timer Unit with the same name as the service we want to start does exactly that!

# /etc/systemd/system/backup.timer
[Unit]
Description=Backup daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

This simple timer is set to run daily at midnight. Setting the Persistent attribute tells systemd that if our host was down at midnight, it has to run the backup as soon as it comes up again. As servers are usually online 247 this doesn’t really matter but it is a great setting for desktops and laptops.

Enable the timer using systemctl:

systemctl enable backup.timer

Now enjoy having a fresh backup of your server every night! Similar to the backup service, you can also create a service to remove your old snapshots. By default Restic just keeps adding data and never removes anything, this may become a problem pretty fast if you’re backup up a lot of data.

Restoring

As with any backup system: remember to test restoring your backups regularly. Restic gives us two great options in case we fuck up and need our data back: