Differences
This shows you the differences between two versions of the page.
Next revision | Previous revision | ||
scripts:backups_and_data [2021-05-01 03:01] – created tony | scripts:backups_and_data [2022-06-03 20:42] (current) – technotony | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | Tony's backup, archiving | + | ====== Backups |
- | coming soon. | ||
+ | ===== Backups ===== | ||
+ | |||
+ | ==== Borg ==== | ||
+ | |||
+ | [[scripts: | ||
+ | |||
+ | ===== Parity ===== | ||
+ | |||
+ | [[scripts: | ||
+ | |||
+ | ===== Compression ===== | ||
+ | |||
+ | ==== XZ/ZSTD ==== | ||
+ | |||
+ | |||
+ | I used to use XZ for compressing tar files. As of 2021-01, I have switched to ZStandard since it offers the same or slightly better ratio with -22, with MUCH faster decompression. | ||
+ | |||
+ | I will try to do a larger writeup and comparison in the future. | ||
+ | |||
+ | For compressing large amounts of files for archiving, where I won't need frequent random access, I use zpaq -m4. m5 is better but also takes forever. ZPAQ is particularly good if there are lots of duplicates. | ||
+ | |||
+ | For folders with large duplicates I rarely need, I am testing out dwarfs, since it hits a good ratio with -l7 (which uses zstd -22) and allows random access. Anything above -7 segfaults for me. | ||
+ | |||
+ | |||
+ | ==== ZPAQ ==== | ||
+ | |||
+ | If you are low on space and don't care about CPU time, it's a journaled archive. Very good compression ratio. | ||
+ | |||
+ | -m3 works fine for maybe-already compressed media like photos and videos you don't want to spend a year on. -m4 is great for more text based files. | ||
+ | |||
+ | zpaq a files.zpaq Documents/ -m4 | ||
+ | | ||
+ | -m5 takes an eternity, and is technically better but generally not worth it. (Bonus points for the RAM usage on a 16c32t system...) | ||
+ | |||
+ | ==== Dwarfs and Squashfs ==== | ||
+ | |||
+ | These are read-only filesystems. you make them, and it's a compressed file, but it can be mounted. | ||
+ | |||
+ | Squashfs: pretty standard. used on many linux installer. similar ratio to tar. can use zstandard. not best radio, but deduplicated, | ||
+ | |||
+ | For backup, borg also allows mounting, is deduplicated and compressed. Sqsh is just a "read only image" equivalent with compression. | ||
+ | |||
+ | Dwarfs: found it on github (by mhx?). faster then zpaq, better then squashfs. have not vetted source code though. use at your own risk. it is a cool FS though. | ||
+ | |||
+ | |||
+ | ===== Other random tools ===== | ||
+ | |||
+ | ==== fpart ==== | ||
+ | |||
+ | This is something I didn't know existed until I tried doing it manually. Basically, say you have 5000 photos to back up. | ||
+ | |||
+ | Say you want to send them to older family members with DVDs (yes, I know, optical media in 2021, give me a break). | ||
+ | |||
+ | DVDs can fit 4.4 GiB each, so you want to take advantage of that and split the files evenly. Doing it by hand sucks. | ||
+ | |||
+ | fpart can take the folders and make file lists for several partitions, either by target number of partitions, OR, by space. | ||
+ | |||
+ | eg: to split into 5 partitions with the file lists named list.0, so on | ||
+ | fpart -n 5 -o list -v . | ||
+ | | ||
+ | Problem is, it doesn' | ||
+ | sed 's/^ *//' < list.0 | xargs -d ' | ||
+ | | ||
+ | Where list.0 is the text list, and folder0 is the target. | ||
+ | |||
+ | The --backup=t makes sure that if there are duplicate file names, it will rename one automatically and not overwrite. | ||
+ | |||
+ | NOTE: THIS REMOVES THE DIRECTORY STRUCTURE, AND LUMPS EVERYTHING IN ONE FOLDER. | ||
+ | |||
+ | This is easy enough to then burn onto a CD/DVD with K3B. While you're at it, do yourself a favor and use PAR2 to add parity. | ||
+ | |||
+ | DVDs will get scratched and I currently leave ~5% parity to recover files. |