Differences

This shows you the differences between two versions of the page.

--- scripts:backups_and_data [2021-05-08 23:17] – tony
+++ scripts:backups_and_data [2022-06-03 20:42] (current) – technotony
@@ Line 1: / Line 1: @@
-===== Backups =====
+====== Backups and Data ======
-I use Borg for my current active backup on my laptop. Since the laptop is only schoolwork, there isn't a ton of other files to copy.
-Borg has a few cool features:
+===== Backups =====
-  * Incremental backups only copy the differences
-  * Ability to mount the backups easily for recovery
-For my desktop, I use good old tar files generally for system setup. All important data on this system is generally copied to one of many external drives.
+==== Borg ====
-I used to use Syncthing, to sync files across my systems. I later switched to unison over ssh through my own VPN. I now just use rsync for simplicity and speed.
+[[scripts:backups:borg|Borg]]
-I have various drives, some old, some new holding backups. Critical things like family documents are redundant on my drive, at least 2 externals, and an encrypted copy on Backblaze B2.
+===== Parity =====
-Old drives with bad partitions are generally retired and become permanent backups after I load an important collection and leave it. I try to power them up twice a year to ensure they still work.
+[[scripts:backups:parity_par2|Parity and Par2]]
-I use PAR2 to create parity files. I do this for both TAR and ZPAQ. I have 2 options: use files/folders and if bitrot happens, only lose 1 file, or use archives/containers, where bitrot can kill everything. However, using parity with 5% prevention should be enough to protect the entire archive.
+===== Compression =====
-If I have to encrypt anything, I use AES256 with GPG. If there is a fault in AES, lets be real, everyone will be screwed anyways.
+==== XZ/ZSTD ====
-===== Compression =====
 I used to use XZ for compressing tar files. As of 2021-01, I have switched to ZStandard since it offers the same or slightly better ratio with -22, with MUCH faster decompression.
@@ Line 31: / Line 26: @@
+==== ZPAQ ====
+If you are low on space and don't care about CPU time, it's a journaled archive. Very good compression ratio.
+-m3 works fine for maybe-already compressed media like photos and videos you don't want to spend a year on. -m4 is great for more text based files.
+  zpaq a files.zpaq Documents/ -m4
+-m5 takes an eternity, and is technically better but generally not worth it. (Bonus points for the RAM usage on a 16c32t system...)
+==== Dwarfs and Squashfs ====
+These are read-only filesystems. you make them, and it's a compressed file, but it can be mounted.
+Squashfs: pretty standard. used on many linux installer. similar ratio to tar. can use zstandard. not best radio, but deduplicated, and allows random reads since it can be mounted.
+For backup, borg also allows mounting, is deduplicated and compressed. Sqsh is just a "read only image" equivalent with compression.
+Dwarfs: found it on github (by mhx?). faster then zpaq, better then squashfs. have not vetted source code though. use at your own risk. it is a cool FS though.
+===== Other random tools =====
+==== fpart ====
+This is something I didn't know existed until I tried doing it manually. Basically, say you have 5000 photos to back up.
+Say you want to send them to older family members with DVDs (yes, I know, optical media in 2021, give me a break).
+DVDs can fit 4.4 GiB each, so you want to take advantage of that and split the files evenly. Doing it by hand sucks.
+fpart can take the folders and make file lists for several partitions, either by target number of partitions, OR, by space.
+eg: to split into 5 partitions with the file lists named list.0, so on
+  fpart -n 5 -o list -v .
+Problem is, it doesn't really move the files. If you don't care about folder structure (ie: intended as slideshow or something)
+  sed 's/^ *//' < list.0 | xargs -d '\n' mv --backup=t -v -t folder0
+Where list.0 is the text list, and folder0 is the target.
+The --backup=t makes sure that if there are duplicate file names, it will rename one automatically and not overwrite.
+NOTE: THIS REMOVES THE DIRECTORY STRUCTURE, AND LUMPS EVERYTHING IN ONE FOLDER.
+This is easy enough to then burn onto a CD/DVD with K3B. While you're at it, do yourself a favor and use PAR2 to add parity.
+DVDs will get scratched and I currently leave ~5% parity to recover files.