My backup strategy follows the 3-2-1 backup rule:
\nSome people argue that your production data (the copy stored on your laptop, for example) counts as one of those copies.\nI disagree.\nWhile it may be a copy of your current data, it's not a copy of your historical data.\nIf I need to recover a file that I modified or deleted 2 years ago, my laptop won't have it.
\nI use Restic to write encrypted backups to a 2.5" SSD inside an external USB enclosure.\nThis disk also contains a few scripts to automate the process.\nFriday is backup day: connect the disk, run the script, done.
\nUsing rsync, I make copies of this master disk:
To another USB 2.5" hard drive located at the office.\nThis is a physical copy I can easily access if my house burns down.\nRestic's encryption is vital here, as coworkers have access to the area where this disk is kept.
\nTo a dedicated server at Hetzner.\nA few friends have SSH access1 to the account that stores this copy.\nThis provides another "worst-case scenario" path to my data.
\nEvery few months I run a validation script on my Hetzner server.\nThis executes Restic with the check --read-data subcommand on the copy stored there.\nA few days later, I know whether the data is still correct.\nWhile not exactly the same as a full restore test, it gives me some piece of mind.
Important folders on my desktop and laptop are synced using Syncthing.\nIn addition to my personal devices, these also replicate to one of my servers.\nFrom there, an hourly Restic snapshot is made to Backblaze B2.\nIn case my 3-2-1 drives fail, I have one last chance to recovery my important documents here.
\nI was assured my backup scheme was solid, until it wasn't.
\nSuddenly, the data validation on the Hetzner server failed.2\nRestic saves backups as deduplicated chunks in what it calls "pack files."\nSeveral of these files no longer matched their checksums.\nSince pack files are named after their SHA-256 hash, I checked them again manually.\nSadly, sha256sum confirmed the worst: the data in these files was not what it was supposed to be.
This server has two 4TB disks in RAID 1.\nI assumed there was a hardware issue with the server, but everything looked fine.\nThe other services on the server ran fine, the disks reported no errors, and the RAID was healthy.
\nThen I realized: the Hetzner server was not the source of corruption.\nChecking the pack files on my USB disk quickly revealed the truth.\nThese files were also corrupt, and rsync had dutifully replicated the corrupted data to my server.
It quickly became obvious what was happening.\nDuring additional checks on my USB disk, my laptop started reporting I/O errors.\nThe SSD in this enclosure has been failing silently for a while, returning random data instead of my files.\nI hadn't noticed because I rarely use these backups.\nThe data is usually only read by rsync to push it to the server.
\nAn hour later, the SSD died completely.
\nTwo days later, when I was back at the office, I checked the second USB drive.\nI had little hope.\nI usually sync both the Hetzner server and this disk at the same time.\nAs expected, the same errors were present.
\nI had managed to corrupt all three copies of my backups simultaneously.
\nThere is no magic recovery procedure here.\nIf all copies of a piece of data are gone, they are gone forever.\nRegrettably, I did lose some data that day.
\nHowever, Restic was quite helpful during the recovery process:
\nA nice side-effect of the data deduplication is that you can "import" files again.\nBy creating a temporary snapshot with files gathered from other locations (like old laptop folders or emails), Restic recreates the deduplicated chunks.\nThis makes those files usable again in the older, previously "corrupt" snapshots!
\nDry-running restic repair snapshots --forget listed exactly which files would be removed due to missing data.\nUse this as a to-do list to gather missing files.
Part of the missing data turned out to be logfiles I didn't care about.\nPurging these from snapshots made it a lot easier to identify which important files were actually missing.3
\nIn the end, I lost only three possibly important files: pictures.\nSince the data is gone, I don't know exactly what was in them, but I know which albums they belonged to.\nMy backup disk now contains a text file listing those three missing filenames as a permanent reminder.
\nClearly, I needed safeguards.\nI made two specific changes to the script I use to sync backups from my local USB SSD to the other destinations:
\nBefore executing rsync, the script now runs restic check.\nWhile a full --read-data takes too long for a weekly routine, a basic check catches structural errors.\nBefore my previous SSD died, the basic check also found errors.\nI'm not sure it would have catched the SSD returning small bits of random data, though.
Except for the locks directory, restic never overwrites files.\nIt only creates new ones.\nOld files stay until you run restic forget to clean up your repository.\nI now run rsync with the --ignore-existing flag, as there is no reason to overwrite remote files.
I'm confident that this would prevent data loss if a drive fails silently again.
\nOnce hard drive prices stabilize, I'll likely buy another external drive for a yearly "cold" archive.\nThe hardest files to replace where those that have been removed from my devices years ago.\nHaving an additional offline drive with an old copy would allow me to recover those files.
\nNot all of them know about it though. ↩
\nI would have never discovered about this if it wasn't for this regular data check.\nBackups are not backups if you don't check them. ↩
\nFiguring out what data I had lost took a few days and I didn't really take notes.\nDocumenting which commands I used here would have been the most interesting part of the article.\nOh well.. ↩
\nSince moving away from Digital Ocean (and their snapshots) I've been looking for a good way to create backups for my servers.\nI've recently stumbled upon the Restic project and have been using it ever since.
\nTransIP is giving 1TB of free storage on their Stack storage service, which is more than enough for my backup needs.\nThe downside is that the only useful way to access it is a painfully slow webdav mount.\nIf you're backing up data that changes fast and need a consistent state, I would suggest choosing a different place to store your data.\nFor most of my servers I don't really care about backup speeds, though.
\nMounting the volume using an entry in /etc/fstab would work, but as I want to specify dependencies later I opt to create a Mount Unit instead.\nThis is basically the systemd version of a single fstab entry.
# /etc/systemd/systemd/mnt-backups.mount\n[Unit]\nDescription=Backup storage\nAfter=network-online.target\nWants=network-online.target\n\n[Mount]\nType=davfs\nWhat=https://something.stackstorage.com/remote.php/webdav/\nWhere=/mnt/backups\nOptions=uid=root,gid=root,file_mode=644,dir_mode=755,nosuid,noexec,rw\nTiemoutSec=15\n\n[Install]\nWantedBy=multi-user.target\nIf you're using webdav (using davfs2) like me, you'll also have to set your username and password in the secrets file /etc/davfs2/secrets.\nBe sure to restrict the permissions on this file as it contains passwords.
https://something.stackstorage.com/remote.php/webdav/ username password\nWe can now tell systemd to mount our volume:
\nsystemctl start mnt-backups.mount\nAnd also set it to be mounted every time our machine boots:
\nsystemctl enable mnt-backups.mount\nBefore our first backup we have to initialize a repository.\nThis creates the directory structure and configuration that restic will use to store our backups.\nNote that, as restic does not trust our backup volume, any configuration stored there is also encrypted!
\nrestic init --repo /mnt/backups/myserver\nDo not forget the password you enter during this step.\nEvery single file on the backup volume is encrypted and without the password your backups become useless.
\nThe Service Unit shown below describes a service that creates a single backup and stops.
\n# /etc/systemd/system/backup.service\n[Unit]\nDescription=Backup\nAfter=mnt-backups.mount\nWants=mnt-backups.mount\n\n[Service]\nType=simple\nEnvironment=RESTIC_PASSWORD=mysupers3cr3tpassw0rd!\nEnvironment=RESTIC_REPOSITORY=/mnt/backups/myserver\nExecStart=/usr/local/bin/restic backup /home /srv/web\nIn the [Unit] section we specify that the mount we created earlier is a requirement for making backups (Wants).\nThe After setting is used to order the units when both are set to start when booting the system.\nIt's a bit redundant to specify that the Service should be after the Mount as we already specified a dependency, but it's a common thing to do.
Note that I only backup my /home and /srv/web folders as everything else can either be recreated or is managed by Ansible on my servers.\nIf you prefer to have a copy of your entire filesystem remember to change the path!
We can now create a new backup manually by running systemctl start backup.\nThis isn't really useful by itself as we could have run restic backup manually just as easy, with the exception that we don't have to type the password anymore.\nOnce we combine this with a systemd timer, things become interesting!
Remember that we set the backup volume to be mounted on boot? That isn't really required.\nAs we have specified a dependency on the volume, systemd will check if it's mounted before starting the backup service.\nThe free Stack volume I'm using is a bit wonky though, so I prefer to keep it mounted as that's a bit more stable.
\nNow that our backup works, it's time to have it create a backup automatically every night.\nCreating a Timer Unit with the same name as the service we want to start does exactly that!
\n# /etc/systemd/system/backup.timer\n[Unit]\nDescription=Backup daily\n\n[Timer]\nOnCalendar=daily\nPersistent=true\n\n[Install]\nWantedBy=timers.target\nThis simple timer is set to run daily at midnight.\nSetting the Persistent attribute tells systemd that if our host was down at midnight, it has to run the backup as soon as it comes up again.\nAs servers are usually online 24/7 this doesn't really matter but it is a great setting for desktops and laptops.
Enable the timer using systemctl:
\nsystemctl enable backup.timer\nNow enjoy having a fresh backup of your server every night!\nSimilar to the backup service, you can also create a service to remove your old snapshots.\nBy default Restic just keeps adding data and never removes anything, this may become a problem pretty fast if you're backup up a lot of data.
\nAs with any backup system: remember to test restoring your backups regularly.\nRestic gives us two great options in case we fuck up and need our data back:
\n