Backups and Data

Borg

Parity and Par2

I used to use XZ for compressing tar files. As of 2021-01, I have switched to ZStandard since it offers the same or slightly better ratio with -22, with MUCH faster decompression.

I will try to do a larger writeup and comparison in the future.

For compressing large amounts of files for archiving, where I won't need frequent random access, I use zpaq -m4. m5 is better but also takes forever. ZPAQ is particularly good if there are lots of duplicates.

For folders with large duplicates I rarely need, I am testing out dwarfs, since it hits a good ratio with -l7 (which uses zstd -22) and allows random access. Anything above -7 segfaults for me.

If you are low on space and don't care about CPU time, it's a journaled archive. Very good compression ratio.

-m3 works fine for maybe-already compressed media like photos and videos you don't want to spend a year on. -m4 is great for more text based files.

zpaq a files.zpaq Documents/ -m4

-m5 takes an eternity, and is technically better but generally not worth it. (Bonus points for the RAM usage on a 16c32t system…)

These are read-only filesystems. you make them, and it's a compressed file, but it can be mounted.

Squashfs: pretty standard. used on many linux installer. similar ratio to tar. can use zstandard. not best radio, but deduplicated, and allows random reads since it can be mounted.

For backup, borg also allows mounting, is deduplicated and compressed. Sqsh is just a “read only image” equivalent with compression.

Dwarfs: found it on github (by mhx?). faster then zpaq, better then squashfs. have not vetted source code though. use at your own risk. it is a cool FS though.

This is something I didn't know existed until I tried doing it manually. Basically, say you have 5000 photos to back up.

Say you want to send them to older family members with DVDs (yes, I know, optical media in 2021, give me a break).

DVDs can fit 4.4 GiB each, so you want to take advantage of that and split the files evenly. Doing it by hand sucks.

fpart can take the folders and make file lists for several partitions, either by target number of partitions, OR, by space.

eg: to split into 5 partitions with the file lists named list.0, so on

fpart -n 5 -o list -v .

Problem is, it doesn't really move the files. If you don't care about folder structure (ie: intended as slideshow or something)

sed 's/^ *//' < list.0 | xargs -d '\n' mv --backup=t -v -t folder0

Where list.0 is the text list, and folder0 is the target.

The –backup=t makes sure that if there are duplicate file names, it will rename one automatically and not overwrite.

NOTE: THIS REMOVES THE DIRECTORY STRUCTURE, AND LUMPS EVERYTHING IN ONE FOLDER.

This is easy enough to then burn onto a CD/DVD with K3B. While you're at it, do yourself a favor and use PAR2 to add parity.

DVDs will get scratched and I currently leave ~5% parity to recover files.