Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
scripts:backups_and_data [2021-05-01 03:01] – created tonyscripts:backups_and_data [2022-06-03 20:42] (current) – technotony
Line 1: Line 1:
-Tony's backup, archiving and data hoarding tips.+====== Backups and Data ======
  
-coming soon. 
  
 +===== Backups =====
 +
 +==== Borg ====
 +
 +[[scripts:backups:borg|Borg]]
 +
 +===== Parity =====
 +
 +[[scripts:backups:parity_par2|Parity and Par2]]
 +
 +===== Compression =====
 +
 +==== XZ/ZSTD ====
 +
 +
 +I used to use XZ for compressing tar files. As of 2021-01, I have switched to ZStandard since it offers the same or slightly better ratio with -22, with MUCH faster decompression.
 +
 +I will try to do a larger writeup and comparison in the future.
 +
 +For compressing large amounts of files for archiving, where I won't need frequent random access, I use zpaq -m4. m5 is better but also takes forever. ZPAQ is particularly good if there are lots of duplicates.
 +
 +For folders with large duplicates I rarely need, I am testing out dwarfs, since it hits a good ratio with -l7 (which uses zstd -22) and allows random access. Anything above -7 segfaults for me.
 +
 +
 +==== ZPAQ ====
 +
 +If you are low on space and don't care about CPU time, it's a journaled archive. Very good compression ratio.
 +
 +-m3 works fine for maybe-already compressed media like photos and videos you don't want to spend a year on. -m4 is great for more text based files.
 +
 +  zpaq a files.zpaq Documents/ -m4
 +  
 +-m5 takes an eternity, and is technically better but generally not worth it. (Bonus points for the RAM usage on a 16c32t system...)
 +
 +==== Dwarfs and Squashfs ====
 +
 +These are read-only filesystems. you make them, and it's a compressed file, but it can be mounted.
 +
 +Squashfs: pretty standard. used on many linux installer. similar ratio to tar. can use zstandard. not best radio, but deduplicated, and allows random reads since it can be mounted.
 +
 +For backup, borg also allows mounting, is deduplicated and compressed. Sqsh is just a "read only image" equivalent with compression.
 +
 +Dwarfs: found it on github (by mhx?). faster then zpaq, better then squashfs. have not vetted source code though. use at your own risk. it is a cool FS though.
 +
 +
 +===== Other random tools =====
 +
 +==== fpart ====
 +
 +This is something I didn't know existed until I tried doing it manually. Basically, say you have 5000 photos to back up.
 +
 +Say you want to send them to older family members with DVDs (yes, I know, optical media in 2021, give me a break).
 +
 +DVDs can fit 4.4 GiB each, so you want to take advantage of that and split the files evenly. Doing it by hand sucks.
 +
 +fpart can take the folders and make file lists for several partitions, either by target number of partitions, OR, by space.
 +
 +eg: to split into 5 partitions with the file lists named list.0, so on
 +  fpart -n 5 -o list -v .
 +  
 +Problem is, it doesn't really move the files. If you don't care about folder structure (ie: intended as slideshow or something)
 +  sed 's/^ *//' < list.0 | xargs -d '\n' mv --backup=t -v -t folder0
 +  
 +Where list.0 is the text list, and folder0 is the target. 
 +
 +The --backup=t makes sure that if there are duplicate file names, it will rename one automatically and not overwrite.
 +
 +NOTE: THIS REMOVES THE DIRECTORY STRUCTURE, AND LUMPS EVERYTHING IN ONE FOLDER.
 +
 +This is easy enough to then burn onto a CD/DVD with K3B. While you're at it, do yourself a favor and use PAR2 to add parity.
 +
 +DVDs will get scratched and I currently leave ~5% parity to recover files.
Navigation