Backups I actually test

Servers · 4 min read

For a long time I had backups that ran every night and a feeling that things were fine. The feeling was the only thing I had ever verified.

Three rules

1. Keep at least one copy on a different machine, ideally in a different place.
2. Automate it. If it needs me to remember, it will stop.
3. Restore something on a schedule. This is the one people skip.

A minimal setup

I use restic because it is a single binary and the commands are short. The paths below are examples:

restic -r /mnt/backup/repo init
restic -r /mnt/backup/repo backup /etc /var/www
restic -r /mnt/backup/repo snapshots
restic -r /mnt/backup/repo restore latest --target /tmp/restore-test

Restic asks for a repository password. Store that password somewhere other than the machine being backed up, otherwise the backup is useless on the day you need it.

The restore drill

Once a month I restore a single file from the latest snapshot into a temporary folder and compare its checksum with the original. It takes two minutes. I write the date and the result in a plain text log, one line each time.

A calendar reminder works better than good intentions. The first drill I ran found that one directory had never been included, and I would much rather learn that on an ordinary evening than during an outage.