I have a server running Debian with 24 TB of storage. I would ideally like to back up all of it, though much of it is torrents, so only the ones with low seeders really need backed up. I know about the 321 rule but it sounds like it would be expensive. What do you do for backups? Also if anyone uses tape drives for backups I am kinda curious about that potentially for offsite backups in a safe deposit box or something.
TLDR: title.
Edit: You have mentioned borg and rsync, and while borg looks good, I want to go with rsync as it seems to be more actively maintained. I would like to also have my backups encrypted, but rsync doesn’t seem to have that built in. Does anyone know what to do for encrypted backups?
You must log in or # to comment.
I’ve been using Borg and Hetzner Storage Box. There are some small VPS hosts that actually beat Hetzner’s pricing but I have been happy with Hetzner so am staying there for now. With 24TB of data you could also look at Hetzner’s SX64 dedicated server. It has a 6 core Ryzen cpu and 4x 16TB HDD’s for 81 euro/month. You could set it up as RAID 10 which would give you around 29 TiB of usable storage, and then you also have a fairly beefy processor that you can use for transcoding and stuff like that. You don’t want to seed from it since Hetzner is sticky about complaints that they might get.
Tape drives are too expensive unless you have 100s of TB of data, I think. Hard drives are too unreliable. If you leave one in a closet for a few years, there’s a good chance it won’t spin back up.
Well, I’m just starting with serious backups, AFAIK you only need to backup the data which you can’t replicate.
Low seeded torrents are just hard to get, but not impossible. Personal photos, your notes, any other files generated by you are the ones which need backups.
Ideally you want to backup everything that you didn’t explicitly exclude since otherwise there is always something you forgot.
Well, I have my personal data in a specific folder, everything there is backed up.
General media is in another one, which isn’t included.
I am simple man s I use rsync.
Setup a mergerfs drive pool of about 60 TiB and rsync weekly.
Rsync seems daunting at first but then you realize how powerful and most importantly reliable it is.
It’s important that you try to restore your backups from time to time.
One of the main reasons why I avoid softwares such as Kopia or Borg or Restic or whatever is in fashion:
- they go unmantained
- they are not simple: so many of my frienda struggled restoring backups because you are not dealing with files anymore, but encrypted or compressed blobs
- rsync has an easy mental model and has extremely good defaults
As long as you understand that simply syncing files does not protect against accidental or malicious data loss like incremental backups do.
I also hope you’re not using
--deletebecause I’ve heard plenty of horror stories about the source dir becoming unmounted and rsync happily erasing everything on the target.I used to use rsync for years, thinking just like you, that having plain old files beats having them in fancy obscure formats. I’m switching to Borg nowadays btw, but that’s my choice, you gotta make yours.
rsync can work incrementally, it just takes a bit more fiddling. Here’s what I did. First of all, no automatic
--delete. I did run it every once in a while but only manually. The sync setup was:- Nightly sync source into nightly dir.
- Weekly sync nightly dir into weekly dir.
- Monthly tarball the weekly dir into monthly dir.
It’s not bad but limited in certain ways, and of course you need lots of space for backups — or you have to pick and choose what you backup.
Borg can’t really get around the space for backups requirement, but it’s always incremental and between compression and deduplication can save you a ton of space.
Borg also has built-in backup checking and recovery parity which rsync doesn’t, you’d have to figure out your own manual solution like par2 checksums (and those take up space too).
Re needing lots of space: you can use --link-dest to make a new directory with hard links to unchanged files in a previous backup. So you end up with de-duplicated incremental backups. But borg handles all that transparently, with rsync you need to carefully plan relative target directory paths to get it to work correctly.