Since moving away from Digital Ocean (and their snapshots) I've been looking for a good way to create backups for my servers.\nI've recently stumbled upon the Restic project and have been using it ever since.
\nTransIP is giving 1TB of free storage on their Stack storage service, which is more than enough for my backup needs.\nThe downside is that the only useful way to access it is a painfully slow webdav mount.\nIf you're backing up data that changes fast and need a consistent state, I would suggest choosing a different place to store your data.\nFor most of my servers I don't really care about backup speeds, though.
\nMounting the volume using an entry in /etc/fstab would work, but as I want to specify dependencies later I opt to create a Mount Unit instead.\nThis is basically the systemd version of a single fstab entry.
# /etc/systemd/systemd/mnt-backups.mount\n[Unit]\nDescription=Backup storage\nAfter=network-online.target\nWants=network-online.target\n\n[Mount]\nType=davfs\nWhat=https://something.stackstorage.com/remote.php/webdav/\nWhere=/mnt/backups\nOptions=uid=root,gid=root,file_mode=644,dir_mode=755,nosuid,noexec,rw\nTiemoutSec=15\n\n[Install]\nWantedBy=multi-user.target\nIf you're using webdav (using davfs2) like me, you'll also have to set your username and password in the secrets file /etc/davfs2/secrets.\nBe sure to restrict the permissions on this file as it contains passwords.
https://something.stackstorage.com/remote.php/webdav/ username password\nWe can now tell systemd to mount our volume:
\nsystemctl start mnt-backups.mount\nAnd also set it to be mounted every time our machine boots:
\nsystemctl enable mnt-backups.mount\nBefore our first backup we have to initialize a repository.\nThis creates the directory structure and configuration that restic will use to store our backups.\nNote that, as restic does not trust our backup volume, any configuration stored there is also encrypted!
\nrestic init --repo /mnt/backups/myserver\nDo not forget the password you enter during this step.\nEvery single file on the backup volume is encrypted and without the password your backups become useless.
\nThe Service Unit shown below describes a service that creates a single backup and stops.
\n# /etc/systemd/system/backup.service\n[Unit]\nDescription=Backup\nAfter=mnt-backups.mount\nWants=mnt-backups.mount\n\n[Service]\nType=simple\nEnvironment=RESTIC_PASSWORD=mysupers3cr3tpassw0rd!\nEnvironment=RESTIC_REPOSITORY=/mnt/backups/myserver\nExecStart=/usr/local/bin/restic backup /home /srv/web\nIn the [Unit] section we specify that the mount we created earlier is a requirement for making backups (Wants).\nThe After setting is used to order the units when both are set to start when booting the system.\nIt's a bit redundant to specify that the Service should be after the Mount as we already specified a dependency, but it's a common thing to do.
Note that I only backup my /home and /srv/web folders as everything else can either be recreated or is managed by Ansible on my servers.\nIf you prefer to have a copy of your entire filesystem remember to change the path!
We can now create a new backup manually by running systemctl start backup.\nThis isn't really useful by itself as we could have run restic backup manually just as easy, with the exception that we don't have to type the password anymore.\nOnce we combine this with a systemd timer, things become interesting!
Remember that we set the backup volume to be mounted on boot? That isn't really required.\nAs we have specified a dependency on the volume, systemd will check if it's mounted before starting the backup service.\nThe free Stack volume I'm using is a bit wonky though, so I prefer to keep it mounted as that's a bit more stable.
\nNow that our backup works, it's time to have it create a backup automatically every night.\nCreating a Timer Unit with the same name as the service we want to start does exactly that!
\n# /etc/systemd/system/backup.timer\n[Unit]\nDescription=Backup daily\n\n[Timer]\nOnCalendar=daily\nPersistent=true\n\n[Install]\nWantedBy=timers.target\nThis simple timer is set to run daily at midnight.\nSetting the Persistent attribute tells systemd that if our host was down at midnight, it has to run the backup as soon as it comes up again.\nAs servers are usually online 24/7 this doesn't really matter but it is a great setting for desktops and laptops.
Enable the timer using systemctl:
\nsystemctl enable backup.timer\nNow enjoy having a fresh backup of your server every night!\nSimilar to the backup service, you can also create a service to remove your old snapshots.\nBy default Restic just keeps adding data and never removes anything, this may become a problem pretty fast if you're backup up a lot of data.
\nAs with any backup system: remember to test restoring your backups regularly.\nRestic gives us two great options in case we fuck up and need our data back:
\nSystemd is mainly used to manage services on modern linux distributions, but it also allows non-root users to manage services running under their own account.\nThis makes systemd a great alternative to services like supervisord as it is able to detect crashes and automatically restarts the service.
\nThe directory structure that holds unit files is similar to the systemwide configuration, but is located in a user's home directory:
\n$ mkdir -p ~/.config/systemd/user\nIn the user folder we can create service units with a .service extension. A unit is a ini-style file describing a service that has to be managed.
Let's say we have a Laravel PHP queue worker that we want to be managed by systemd, called worker.service:
[Unit]\nDescription=Laravel PHP queue worker\n\n[Service]\nExecStart=/usr/bin/env php /path/to/my/project/artisan queue:work --sleep=3 --tries=3\n\n[Install]\nWantedBy=default.target\nRun $ systemctl --user daemon-reload to make systemd recognize your changes.
We can now manage this service with systemd:
\n$ systemctl --user start worker starts the service$ systemctl --user stop worker stops the service$ systemctl --user restart worker restarts the workers$ systemctl --user status worker shows us the service statusIf we want this service to start automatically when our user logs in, we have to enable it:
\n$ systemctl --user enable worker$ systemctl --user disable workerEverything our PHP script writes to stdout and stderr will be recorded by journald. We can view the logs for this specific service:
\n$ journalctl -u worker\nOn older versions of systemd (like the one shipped with Debian Jessie), you might need to use a flag specific to user units:
\n$ journalctl --user-unit worker\nLet's be honest: things go wrong. When our worker crashes we want to make sure it's started again. Fortunately, we can configure systemd to restart on failure:
\nAdd the following to the [Service] section of the unit file:
RestartSec=10s\nRestart=on-failure\nThis makes systemd restart the service 10 seconds after it has stopped with a non-zero exit code. If the process exited cleanly, systemd considers the work as done and won't restart the service until you manually start it later.
\nBy default, systemd starts enabled user units when the user logs in, and stops them once the user has logged out. This is great for desktop pcs where a user wants to run things while he's using the computer. On servers, however, we may want to run those units all the time.
\n# loginctl enable-linger USERNAME\nYou can check the status for each user by:
\n# loginctl user-status USERNAME\nIt should show that the user is lingering:
\nvic (1000)\n Since: Fri 2017-09-01 21:54:29 CEST; 3s ago\n State: lingering\n Unit: user-1000.slice\nWhen the user is not logged in and does not have lingering enabled, loginctl doesn't know about any user sessions so it will return an error:
\nFailed to get user: No user 1000 known or logged in\n