Something to consider, advice given to me, is that ZFS support on Linux regularly breaks with newest kernels so if you go for ZFS long term, be prepared to run a lts kernel at least as a backup.

I use both. LUKS+btrfs being nice on the Arch desktops, and ZFS on a serverside pool, managed by a TrueNAS Scale VM.

Oh yeah and I did enable Proxmox VM firewall for the TrueNAS, the NFS traffic goes via an internal interface. Wasn’t entirely convinced by NFS’s security posture when reading about it… At least restrict it to the physical machine 0_0 So I now need to intentionally pass a new NIC to any VM that will access the data, which is neat.

A wrap-up of what I ended up doing:

• Replaced the bare metal Ubuntu with Proxmox. Cool cool. It can do the same stuff but easier / comes with a lot of hints for best practices. Guess I’m a datacenter admin now
• Wiped the 2x960GB SSD pool and re-created it with ZFS native encryption
• Made a TrueNAS Scale VM, passed through the SSD pool disks, shared the datasets with NFS and made snapshot policies
• Mounted the NFS on the Ubuntu VM running my data related services and moved the docker bind mounts to that folder
• Bought a 1Gbps Intel network card to use instead of the onboard Realtek and maxed out the host memory to 16GB for good measure

I have achieved:

• 15min RPO for my data (as it sits on the NFS mount, which is auto-snapshotted in TrueNAS)
• Encryption at rest (ZFS native)

I have not achieved (yet…):

• Key fetch on boot. Now if the host machine boots I have to log in to TrueNAS to key in the ZFS passphrase. I will have to make some custom script for this anyway I guess to make it adapt to the situation as key fetching on boot is a paid feature in TrueNAS but it just makes managing the storage a bit easier so I wanna use it now. Disabled auto start on boot for the services VM that depends on the NFS share, so I’ll just go kick it up manually after unlocking the pool in TrueNAS.

Quite happy with the setup so far. Looking to automate actual backups next, but this is starting to take shape. Building the confidence to use this for my actual phone backups, among other things.

Really good to know. Planned to keep using very mainstream LTS versions anyway, but this solidifies the decision. Maybe on a laptop I’ll install something more experimental but that’s then throwaway style.

I guess I’ll give it a spin. There seems to be a big community around it. I initially thought I might migrate later so keeping the host OS layer as thin as possible. Ubuntu was mainly an easy start as I was familiar with it from before and the spirit in this initiative is DIY over framework - but if there’s a widely used solution for exactly this… Yeah.

Always a good reminder to test the backups, no I would not sleep properly if I didn’t test them :p

Aiming to keep it simple, too many moving parts in the VM snapshots / hard to figure out best practices and notice mistakes without work experience in the area, so I’ll just backup the data separately and call it a day. But thanks for the input! I don’t think any of my services have in-memory db’s.

Right, thanks for the heads up! On the desktops I have simply installed zfs as root via the Ubuntu 24.04 installer. Then, as the option was not available in the server variant I started to think maybe that is not something that should be done :p

Aight thank you so much, confirms I’m on the right path! This clarifies a lot, I’ll keep the ext4 boot drive :)

Right, so my aversion to live backups comes initially from Louis Rossmann’s guide on the FUTO wiki where he mentions it’s non trivial to reliably snapshot a running system. After a lot of looking elsewhere as well I haven’t gotten much hints that it would be bad advice and I want to err on the side of caution anyway. The hypervisor is QEMU/KVM so in theory it should be able to do live snapshots afaik. But I’m not familiar enough with the consistency guarantees to fully trust it. I don’t wanna wake up one day to a server crash and trying to mount the backed up qcow2 in a new system and suddenly it wouldn’t work and I just lost data.

It won’t matter though as I’ll just place all the important data on the zpool and back that up frequently as a simple data store. The VMs can keep doing their nightly shutdown and snapshot thing.

Ok so wrapping my head around this, what I think I need to be clear about is the separation between applications and data. Applications get the nightly VM snapshot way of backing up, and data will get the frequent zfs snapshots (and other backups). Kinda what I tried to do to begin with, so I will look more on how to do this separation for the applications I intend to use.

Still unsure if samba is the way to go for linking it together on the same physical machine.

Should I just run syncthing on the bare metal host…? Will sleep on it.

This is what I’m doing currently, but it’s not really feasible to have the services shut down hourly for snapshots. This is indeed why I started looking towards filesystem-level snapshotting Obviously I will have other types of backups as well, I’m simply looking to have the on-the-fly immutable snapshot capability here somehow.

Thanks! Can I ask what is your setup like? ZFS on bare metal? Do you have VMs?

Fair about the SSD life. How would you go about achieving the frequent backups without zfs? I wouldn’t want to implement it separately for every app I use, though I’m open to it if this doesn’t work out.

I’ll easily buy more memory if needed, the box now has 8GB and isn’t struggling in any way.

I personally know three, myself included, who are switching right now.