I got 4 20TB drives from Amazon around Black Friday that I want to get setup for network storage. I’ve got 3 descent Ryzen 5000 series desktops that I was thinking about setting up so that I could build my own mini-Kubernetes cluster, but I don’t know if I have enough motivation. I’m pretty OCD so small projects often turn into big projects.

I don’t have an ECC motherboard though, so I want to get some input if BTRFS, ZFS, TrueNAS, or some other solution should be relatively safe without it? I guess it is a risk-factor but I haven’t had any issues yet (fingers crossed). I’ve been out of the CNCF space for a while but Rook used to be the way to go for Ceph on Kubernetes. Has there been any new projects worth checking out or should I just do RAID and get it over with? Does Ceph offer the same level of redundancy or performance? The boards have a single M.2 slot so I could add in some SSD caching.

If I go with RAID, should I do RAID 5 or 6? I’m also a bit worried because the drives are all the same so if there is an issue it could hit multiple drives at once, but I plan to try to have an online backup somewhere and if I order more drives I’ll balance it out with a different manufacturer.

  • terribleplan@lemmy.nrd.liEnglish
    01·
    1 year ago

    Based on the hardware you have I would go with ZFS (using TrueNAS would probably be easiest). Generally with such large disks I would suggest using at least 2 parity disks, but seeing as you only have 4 that means you would lose half your storage to be able to survive 2 disk failures. Reason for the (at least) 2 parity disks is (especially with identical disks) the risk of failure during a rebuild after one failure is pretty high since there is so much data to rebuild and write to your new disk (like, it will probably take more than a day).

    Can’t talk much about backup as I just have very little data that I care enough about to backup, and just throw that into cloud object storage as well as onto my local high-reliability storage.

    I have tried many different solutions, so will give you a quick overview of my experiences, thoughts, and things I have heard/seen:

    Single Machine

    Do this unless you have to scale beyond one machine

    ZFS (on TrueNAS)

    • It’s great, with a few exceptions.
    • Uses data checksums so it can detect bitrot when performing a “scrub”.
    • Super simple to manage, especially with the FreeNAS TrueNAS GUI.
    • Can run some services via Jails and/or plugins
    • It only works on a single machine, which became a limiting factor for me.
    • It can’t add disks one at a time, you have to add an entire vdev (another set of drives in RAID-Z or whatever you choose).
    • You have to upgrade all disks in a vdev to use higher capacity disks.
    • Has lots of options for how to use disks in vdevs:
      • Stripe (basically RAID-0, no redundancy, only for max performance)
      • Mirror (basically RAID-1, same data on every disk in the vdev)
      • RAID-Zx (basically RAID-5, RAID-6, or <unnamed raid level better than 6>, uses x # of disks for parity, meaning that many disks can be lost)
    • ZFS send seems potentially neat for backups, though I have never used it

    MDADM

    • It’s RAID, just in your linux kernel.
    • Has been in the kernel for years, is quite reliable. (I’ve been using it for literally years on a few different boxes as ZFS on Linux was less mature at the time)
    • You can make LVM use it mostly transparently.
    • I would probably run ZFS for new installs instead.

    BTRFS

    • Can’t speak to this one with personal experience.
    • Have heard it works best on SSDs, not sure if that is the case any more.
    • The RAID offerings used to be questionable, pretty sure that isn’t the case any more.

    UnRaid

    • It’s a decently popular option.
    • It lets you mix disks of different capacity, and uses your largest disk for parity
    • Can just run docker containers, which is great.
    • Uses a custom solution for parity, so likely less battle-hardened and less eyes on it vs ZFS or MDAM.
    • Parity solution reminds me of RAID-4, which may mean higher wear on your parity drive in some situations/workloads.
    • I think they added support for more than one parity disk, so that’s neat.

    Raid card

    • Capabilities and reliability can vary by vendor
    • Must have battery backup if you are using write-back for performance gains
    • Seemingly have fallen out of favor to JBODs with software solutions (ZFS, BTRFS, UnRaid, MDADM)
    • I use the PERCs in my servers for making a RAID-10 pool out of local 2.5in disks on some of my servers. Works fine, no complaints.

    JBOD

    • Throwing this in here as it is still mostly one machine, and worth mentioning
    • You can buy basically a stripped down server (just a power supply and special SAS expander card) that you can put disks in, and lets you connect that shelf of storage to your actual server
    • May let you scale some of those “Single Machine” solutions beyond the number of drive bays you have.
    • Is putting a number of eggs in one basket as far as hardware goes if the host server dies, up to you to decide how you want to approach that.

    Multi-machine

    Ceph

    • Provides block (RBD), FS, and Object (S3) storage interfaces.
    • Used widely by cloud providers
    • Companies I’ve seen run it often have a whole (small) team just to build/run/maintain it
    • I had a bad experience with it
      • Really annoying to manage (even with cephadm)
      • Broke for unclear reasons, while appearing everything was working
      • I lost all the data I put into during testing
      • My experience may not be representative of what yours would be

    SeaweedFS

    • Really neat project
    • Combines some of the best properties of replication and erasure coding
      • Stores data in volume files of X size
      • Read/Write happens on replica volumes
      • Once a volume fills you can set it as read only and convert it to erasure coding for better space efficiency
      • This can make it harder to reclaim disk space, so depending on your workload may bot be right for you
    • Has lots of storage configuration options for volumes to tolerate machine/rack/row failures.
    • Can shift cold data to cloud storage and I think even can back itself up to cloud storage
    • Can provide S3, WebDAV, and FUSE storage natively
    • Very young project
    • Management story is not entirely figured out yet
    • I also lost data while testing this, though root cause there was unreliable hardware

    Tahoe LAFS

    • Very brief trial
    • Couldn’t wrap my head around it
    • Seems interesting
    • Seems mostly designed for storing things reliably on untrusted machines, so my use case was probably not ideal for it.

    MooseFS/LizardFS

    • Looked neat and had many of the features I want
    • Some of those features are only on (paid) MooseFS Pro or LizardFS (seemingly abandoned/unmaintained)

    Gluster

    • Disks can be put into many different volume configurations depending on your needs
      • Distributed (just choose a disk for each file, no redundancy)
      • Replicated (store every file on every disk, very redundant, wastes lots of space, only as much space as the smallest disk)
      • Distributed Replicated (Distributed across Replicated sets, add X disks as a Replicated set of disks, choose one of the replica sets and store the file on every disk in that set, is how you scale Replicated disks, each replica can only be as big as the smallest member disk, you must add X disks at a time)
      • Dispersed (store each file across every disk using X disks for parity, tolerates X disk failures, only as much space as the smallest disk * (number of disks - X), means you are only losing X disks worth of parity)
      • Distributed Dispersed (Distributed across Dispersed sets, add X disks as a Dispersed set of disks with Y parity, choose one of the disperse sets and store each file across its X disks using Y disks for parity, is how you scale Dispersed disks, each disperse only has as much space as the smallest disk * (X - Y), you must add X disks at a time)
    • Also gets used by enterprises
    • Anything but dispersed stores full files on a normal filesystem (vs Ceph using its own special filesystem, vs Seaweed that stores things in volume files) meaning in a worst case recovery scenario you can read the disks directly.
    • Very easy to configure
    • I am using it now, my testing of it went well
      • Jury is still out on what maintenance looks like

    Kubernetes-native

    Consider these if you’re using k8s. You can use any of the single-machine options (or most of the multi-machine options) and find a way to use them in k8s (natively for gluster and some others, or via NFS). I had a lot of luck just using NFS from my TrueNAS storage server in my k8s cluster.

    Rook

    • Uses Ceph under the hood
    • Used it very briefly and it seemed fine.
    • I have heard good things, but am skeptical given my previous experience with Ceph

    Longhorn

    • Project by the folks at Rancher/SUSE
    • Replicates the volume
    • Worked well enough when I was running k8s with some light workloads in it
    • Only seems to provide block storage, which I am much less interested in.

    OpenEBS

    • Never used it myself
    • Only seems to provide block storage, which I am much less interested in.