You might not even like rsync. Yeah it's old. Yeah it's slow. But if you're working with Linux you're going to need to know it. In this video I walk through my favorite everyday flags for rsync. Su...

You might not even like rsync. Yeah it’s old. Yeah it’s slow. But if you’re working with Linux you’re going to need to know it.

In this video I walk through my favorite everyday flags for rsync.

Support the channel:
https://patreon.com/VeronicaExplains
https://ko-fi.com/VeronicaExplains
https://thestopbits.bandcamp.com/

Here’s a companion blog post, where I cover a bit more detail: https://vkc.sh/everyday-rsync

Also, @BreadOnPenguins made an awesome rsync video and you should check it out: https://www.youtube.com/watch?v=eifQI5uD6VQ

Lastly, I left out all of the ssh setup stuff because I made a video about that and the blog post goes into a smidge more detail. If you want to see a video covering the basics of using SSH, I made one a few years ago and it’s still pretty good: https://www.youtube.com/watch?v=3FKsdbjzBcc

Chapters:
1:18 Invoking rsync
4:05 The --delete flag for rsync
5:30 Compression flag: -z
6:02 Using tmux and rsync together
6:30 but Veronica… why not use (insert shiny object here)

  • NuXCOM_90Percent@lemmy.zipEnglish
    45·
    6 months ago

    I would generally argue that rsync is not a backup solution. But it is one of the best transfer/archiving solutions.

    Yes, it is INCREDIBLY powerful and is often 90% of what people actually want/need. But to be an actual backup solution you still need infrastructure around that. Bare minimum is a crontab. But if you are actually backing something up (not just copying it to a local directory) then you need some logging/retry logic on top of that.

    At which point you are building your own borg, as it were. Which, to be clear, is a great thing to do. But… backups are incredibly important and it is very much important to understand what a backup actually needs to be.

    • tal@olio.cafeEnglish
      16·
      6 months ago

      I would generally argue that rsync is not a backup solution.

      Yeah, if you want to use rsync specifically for backups, you’re probably better-off using something like rdiff-backup, which makes use of rsync to generate backups and store them efficiently, and drive it from something like backupninja, which will run the task periodically and notify you if it fails.

      rsync: one-way synchronization

      unison: bidirectional synchronization

      git: synchronization of text files with good interactive merging.

      rdiff-backup: rsync-based backups. I used to use this and moved to restic, as the backupninja target for rdiff-backup has kind of fallen into disrepair.

      That doesn’t mean “don’t use rsync”. I mean, rsync’s a fine tool. It’s just…not really a backup program on its own.

      • melfie@lemy.lolEnglish
        71·
        6 months ago

        Having a synced copy elsewhere is not an adequate backup and snapshots are pretty important. I recently had RAM go bad and my most recent backups had corrupt data, but having previous snapshots saved the day.

        • melfie@lemy.lolEnglish
          3·
          6 months ago

          Don’t understand the downvotes. This is the type of lesson people have learned from losing data and no sense in learning it the hard way yourself.

          • tomenzgg@midwest.socialEnglish
            2·
            6 months ago

            How would you pin down something like this? If it happened to me, I expect I just wouldn’t understand what’s going on.

            • melfie@lemy.lolEnglish
              1·
              6 months ago

              I originally thought it was one of my drives in my RAID1 array that was failing, but I noticed copying data was yielding btrfs corruption errors on both drives that could not be fixed with a scrub and I was also getting btrfs corruption errors on the root volume as well. I figured it would be quite an odd coincidence if my main SSD and 2 hard disks all went bad and I happened upon an article talking about how corrupt data can also occur if the RAM is bad. I also ran SMART tests and everything came back with a clean bill of health. So, I installed and booted into Memtester86+ and it immediately started showing errors on the single 16Gi stick I was using. I happened to have a spare stick that was a different brand, and that one passed the memory test with flying colors. After that, all the corruption errors went away and everything has been working perfectly ever since.

              I will also say that legacy file systems like ext4 with no checksums wouldn’t even complain about corrupt data. I originally had ext4 on my main drive and at one point thought my OS install went bad, so I reinstalled with btrfs on top of LUKS and saw I was getting corruption errors on the main drive at that point, so it occurred to me that 3 different drives could not have possibly had a hardware failure and something else must be going on. I was also previously using ext4 and mdadm for my RAID1 and migrated it to btrfs a while back. I was previously noticing as far back as a year ago that certain installers, etc. that previously worked no longer worked, which happened infrequently and didn’t really register with me as a potential hardware problem at the time, but I think the RAM was actually progressively going bad for quite a while. btrfs with regular scrubs would’ve made it abundantly clear much sooner that I had files getting corrupted and that something was wrong.

              So, I’m quite convinced at this point that RAID is not a backup, even with the abilities of btrfs to self-heal, and simply copying data elsewhere is not a backup, because something like bad RAM in both cases can destroy data during the copying process, whereas older snapshots in the cloud will survive such a hardware failure. Older data backed up that wasn’t coped with faulty RAM may be fine as well, but you’re taking a chance that a recent update may overwrite good data with bad data. I was previously using Rclone for most backups while testing Restic with daily, weekly, and monthly snapshots for a small subset of important data the last few months. After finding some data that was only recoverable in a previous Restic snapshot, I’ve since switched to using Restic exclusively for anything important enough for cloud backups. I was mainly concerned about the space requirements of keeping historical snapshots, and I’m still working on tweaking retention policies and taking separate snapshots of different directories with different retention policies according risk tolerance for each directory I’m backing up. For some things, I think even btrfs local snapshots would suffice with the understanding that it’s to reduce recovery time, but isn’t really a backup . However, any irreplaceable data really needs monthly Restic snapshots in the cloud. I suppose if don’t have something like btrfs scrubs to alert you that you have a problem, even snapshots from months ago may have an unnoticed problem.

  • mesa@piefed.socialOPEnglish
    32·
    6 months ago

    Ive personally used rsync for backups for about…15 years or so? Its worked out great. An awesome video going over all the basics and what you can do with it.

    • Eager Eagle@lemmy.worldEnglish
      61·
      6 months ago

      It works fine if all you need is transfer, my issue with it it’s just not efficient. If you want a “time travel” feature, your only option is to duplicate data. Differential backups, compression, and encryption for off-site ones is where other tools shine.

      • NekuSoul@lemmy.nekusoul.deEnglish
        1·
        6 months ago

        Agree. It’s neat for file transfers and simple one-shot backups, but if you’re looking for a proper backup solution then other tools/services have advanced virtually every aspect of backups so much it pretty much always makes sense to use one of those instead.

      • suicidaleggroll@lemmy.worldEnglish
        1·
        6 months ago

        If you want a “time travel” feature, your only option is to duplicate data.

        Not true. Look at the --link-dest flag. Encryption, sure, rsync can’t do that, but incremental backups work fine and compression is better handled at the filesystem level anyway IMO.

        • Eager Eagle@lemmy.worldEnglish
          1·
          6 months ago

          Isn’t that creating hardlinks between source and dest? Hard links only work on the same drive. And I’m not sure how that gives you “time travel”, as in, browsing snapshots or file states at the different times you ran rsync.

          Edit: ah the hard link is between dest and the link-dest argument, makes more sense.

          I wouldn’t bundle fs and backup compression in the same bucket, because they have vastly different reqs. Backup compression doesn’t need to be optimized for fast decompression.

      • sugar_in_your_tea@sh.itjust.worksEnglish
        3·
        6 months ago

        That would only matter if it’s lots of small files, right? And after the initial sync, you’d have very few files, no?

        Rsync is designed for incremental syncs, which is exactly what you want in a backup solution. If your multithreaded alternative doesn’t do a diff, rsync will win on larger data sets that don’t have rapid changes.

  • ominous ocelot@leminal.spaceEnglish
    8·
    6 months ago

    rsnapshot is a script for the purpose of repeatedly creating deduplicated copies (hardlinks) for one or more directories. You can chose how many hourly, daily, weekly,… copies you’d like to keep and it removes outdated copies automatically. It wraps rsync and ssh (public key auth) which need to be configured before.

  • 1984@lemmy.todayEnglish
    6·
    6 months ago

    I never thought of it as slow. More like very reliable. I dont need my data to move fast, I need it to be copied with 100% reliability.

  • Landless2029@lemmy.worldEnglish
    5·
    6 months ago

    I need a breakdown like this for Rclone. I’ve got 1TB of OneDrive free and nothing to do with it.

    I’d love to setup a home server and backup some stuff to it.

        • MangoPenguin@lemmy.blahaj.zoneEnglish
          2·
          6 months ago

          What happens if you accidentally overwrite something important in a document and save it though? If there’s no incremental versioning you can’t recover from that.

          • RestrictedAccount@lemmy.worldEnglish
            1·
            6 months ago

            That is a good point.

            In my case, I was trying to address the shortcomings of Apple Time Machine. I use a Mac mini as the server I work from on all my machines. Time Machine does the version Managment for me.

            I just use Sync Thing through a VPN to keep an offsite backup of content files (not a complete OS restore) and to keep a copy of critical files on my laptop in case I am away from my home network and need to see a file.

            I still need to implement a regular air gapped backup instead of the ad-hoc that I have now.

    • Encrypt-Keeper@lemmy.worldEnglish
      6·
      6 months ago

      I’m not super familiar with Syncthing, but judging by the name I’d say Syncthing is not at all meant for backups.

  • surph_ninja@lemmy.worldEnglish
    61·
    6 months ago

    Use borg/borgmatic for your backups. Use rsync to send your differentials to your secondary & offsite backup storage.

  • i_stole_ur_taco@lemmy.caEnglish
    5·
    6 months ago

    The thing I hate most about rsync is that I always fumble to get the right syntax and flags.

    This is a problem because once it’s working I never have to touch it ever again because it just works and keeping working. There’s not enough time to memorize the usage.

    • NuXCOM_90Percent@lemmy.zipEnglish
      3·
      6 months ago

      One trick that one of my students taught me a decade or so ago is to actually make an alias to list the useful flags.

      Yes, a lot of us think we are smart and set up aliases/functions and have a huge list of them that we never remember or, even worse, ONLY remember. What I noticed her doing was having something like goodman-rsync that would just echo out a list of the most useful flags and what they actually do.

      So nine times out of 10 I just want rsync -azvh --progress ${SRC} ${DEST} but when I am doing something funky and am thinking “I vaguely recall how to do this”? dumbman rsync and I get a quick cheat sheet of what flags I have found REALLY useful in the past or even just explaining what azvh actually does without grepping past all the crap I don’t care about in the man page. And I just keep that in the repo of dotfiles I copy to machines I work on regularly.

      • muix@lemmy.sdf.orgEnglish
        1·
        6 months ago

        tldr and atuin have been my main way of remembering complex but frequent flag combinations

        • NuXCOM_90Percent@lemmy.zipEnglish
          1·
          6 months ago

          Yeah. There are a few useful websites I end up at that serve similar purposes.

          My usual workflow is that I need to be able to work in an airgapped environment where it is a lot easier to get “my dotfiles” approved than to ask for utility packages like that. Especially since there will inevitably be some jackass who says “You don’t know how to work without google? What are we paying you for?” because they mostly do the same task every day of their life.

          And I do find that writing the cheat sheet myself goes a long way towards me actually learning them so I don’t always need it. But I know that is very much how my brain works (I write probably hundreds of pages of notes a year… I look at maybe two pages a year).

      • JohnAnthony@lemmy.dbzer0.comEnglish
        1·
        6 months ago

        rsync -avzhP gang unite! I knew someone would have posted my standard flags. I used them enough that my brain moved them from RAM to ROM at this point…

      • tal@olio.cafeEnglish
        1·
        6 months ago

        Most Unix commands will show a short list of the most-helpful flags if you use --help or -h.

    • mesa@piefed.socialOPEnglish
      2·
      6 months ago

      I feel this too. I have a couple of “spells” that work wonders in a literal small notebook with other one liners over the years. Its my spell book lol.

  • tal@olio.cafeEnglish
    4·
    6 months ago

    slow

    rsync is pretty fast, frankly. Once it’s run once, if you have -a or -t passed, it’ll synchronize mtimes. If the modification time and filesize matches, by default, rsync won’t look at a file further, so subsequent runs will be pretty fast. You can’t really beat that for speed unless you have some sort of monitoring system in place (like, filesystem-level support for identifying modifications).

  • Appoxo@lemmy.dbzer0.comEnglish
    4·
    6 months ago

    Veeam for image/block based backups of Windows, Linux and VMs.
    syncthing for syncing smaller files across devices.

    Thank you very much.

  • solrize@lemmy.mlEnglish
    3·
    6 months ago

    I’ve been using borg because of the backend encryption and because the deduplication and snapshot features are really nice. It could be interesting to have cross-archive deduplication but maybe I can get something like that by reorganizing my backups. I do use rsync for mirroring and organizing downloads, but not really for backups. It’s a synchronization program as the name implies, not really intended for backups.