Since moving away from Digital Ocean (and their snapshots) I've been looking for a good way to create backups for my servers. I've recently stumbled upon the Restic project and have been using it ever since.
- Restic is very simple to use. It's a cli-tool with a few easy to understand commands and has good documentation including examples.
- You don't need a special backup service. Restic runs on the host being backed up, and is happy to store its backups on a simple file server or cloud service.
- Security is important and the developers have written a decent threat model: trust only the host where restic runs, which makes sense as it contains the data anyway. The place where you store your backups is always considered insecure.
Mounting storage space
TransIP is giving 1TB of free storage on their Stack storage service, which is more than enough for my backup needs. The downside is that the only useful way to access it is a painfully slow webdav mount. If you're backing up data that changes fast and need a consistent state, I would suggest choosing a different place to store your data. For most of my servers I don't really care about backup speeds, though.
Mounting the volume using an entry in /etc/fstab would work, but as I want to specify dependencies later I opt to create a Mount Unit instead.
This is basically the systemd version of a single fstab entry.
# /etc/systemd/systemd/mnt-backups.mount
[Unit]
Description=Backup storage
After=network-online.target
Wants=network-online.target
[Mount]
Type=davfs
What=https://something.stackstorage.com/remote.php/webdav/
Where=/mnt/backups
Options=uid=root,gid=root,file_mode=644,dir_mode=755,nosuid,noexec,rw
TiemoutSec=15
[Install]
WantedBy=multi-user.target
If you're using webdav (using davfs2) like me, you'll also have to set your username and password in the secrets file /etc/davfs2/secrets.
Be sure to restrict the permissions on this file as it contains passwords.
https://something.stackstorage.com/remote.php/webdav/ username password
We can now tell systemd to mount our volume:
systemctl start mnt-backups.mount
And also set it to be mounted every time our machine boots:
systemctl enable mnt-backups.mount
Initialize the repository
Before our first backup we have to initialize a repository. This creates the directory structure and configuration that restic will use to store our backups. Note that, as restic does not trust our backup volume, any configuration stored there is also encrypted!
restic init --repo /mnt/backups/myserver
Do not forget the password you enter during this step. Every single file on the backup volume is encrypted and without the password your backups become useless.
Backup service
The Service Unit shown below describes a service that creates a single backup and stops.
# /etc/systemd/system/backup.service
[Unit]
Description=Backup
After=mnt-backups.mount
Wants=mnt-backups.mount
[Service]
Type=simple
Environment=RESTIC_PASSWORD=mysupers3cr3tpassw0rd!
Environment=RESTIC_REPOSITORY=/mnt/backups/myserver
ExecStart=/usr/local/bin/restic backup /home /srv/web
In the [Unit] section we specify that the mount we created earlier is a requirement for making backups (Wants).
The After setting is used to order the units when both are set to start when booting the system.
It's a bit redundant to specify that the Service should be after the Mount as we already specified a dependency, but it's a common thing to do.
Note that I only backup my /home and /srv/web folders as everything else can either be recreated or is managed by Ansible on my servers.
If you prefer to have a copy of your entire filesystem remember to change the path!
We can now create a new backup manually by running systemctl start backup.
This isn't really useful by itself as we could have run restic backup manually just as easy, with the exception that we don't have to type the password anymore.
Once we combine this with a systemd timer, things become interesting!
Remember that we set the backup volume to be mounted on boot? That isn't really required. As we have specified a dependency on the volume, systemd will check if it's mounted before starting the backup service. The free Stack volume I'm using is a bit wonky though, so I prefer to keep it mounted as that's a bit more stable.
Creating backups automatically
Now that our backup works, it's time to have it create a backup automatically every night. Creating a Timer Unit with the same name as the service we want to start does exactly that!
# /etc/systemd/system/backup.timer
[Unit]
Description=Backup daily
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
This simple timer is set to run daily at midnight.
Setting the Persistent attribute tells systemd that if our host was down at midnight, it has to run the backup as soon as it comes up again.
As servers are usually online 24/7 this doesn't really matter but it is a great setting for desktops and laptops.
Enable the timer using systemctl:
systemctl enable backup.timer
Now enjoy having a fresh backup of your server every night! Similar to the backup service, you can also create a service to remove your old snapshots. By default Restic just keeps adding data and never removes anything, this may become a problem pretty fast if you're backup up a lot of data.
Restoring
As with any backup system: remember to test restoring your backups regularly. Restic gives us two great options in case we fuck up and need our data back:
- Restore a complete snapshot if you know to which point in time you need to return and if you need a copy of all your files.
- Mount your backup repository as a fuse filesystem, allowing you to browse all your snapshots and the files within as if they're all just there on your server.