Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
scripts:backups_and_data [2021-05-08 23:17] – tony | scripts:backups_and_data [2022-06-03 20:42] (current) – technotony | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ===== Backups ===== | + | ====== Backups |
- | I use Borg for my current active backup on my laptop. Since the laptop is only schoolwork, there isn't a ton of other files to copy. | ||
- | Borg has a few cool features: | + | ===== Backups ===== |
- | * Incremental backups only copy the differences | + | |
- | * Ability to mount the backups easily for recovery | + | |
- | For my desktop, I use good old tar files generally for system setup. All important data on this system is generally copied to one of many external drives. | + | ==== Borg ==== |
- | I used to use Syncthing, to sync files across my systems. I later switched to unison over ssh through my own VPN. I now just use rsync for simplicity and speed. | + | [[scripts: |
- | I have various drives, some old, some new holding backups. Critical things like family documents are redundant on my drive, at least 2 externals, and an encrypted copy on Backblaze B2. | + | ===== Parity ===== |
- | Old drives with bad partitions are generally retired and become permanent | + | [[scripts:backups: |
- | I use PAR2 to create parity files. I do this for both TAR and ZPAQ. I have 2 options: use files/ | + | ===== Compression ===== |
- | If I have to encrypt anything, I use AES256 with GPG. If there is a fault in AES, lets be real, everyone will be screwed anyways. | + | ==== XZ/ZSTD ==== |
- | |||
- | ===== Compression ===== | ||
I used to use XZ for compressing tar files. As of 2021-01, I have switched to ZStandard since it offers the same or slightly better ratio with -22, with MUCH faster decompression. | I used to use XZ for compressing tar files. As of 2021-01, I have switched to ZStandard since it offers the same or slightly better ratio with -22, with MUCH faster decompression. | ||
Line 31: | Line 26: | ||
+ | ==== ZPAQ ==== | ||
+ | |||
+ | If you are low on space and don't care about CPU time, it's a journaled archive. Very good compression ratio. | ||
+ | |||
+ | -m3 works fine for maybe-already compressed media like photos and videos you don't want to spend a year on. -m4 is great for more text based files. | ||
+ | |||
+ | zpaq a files.zpaq Documents/ -m4 | ||
+ | | ||
+ | -m5 takes an eternity, and is technically better but generally not worth it. (Bonus points for the RAM usage on a 16c32t system...) | ||
+ | |||
+ | ==== Dwarfs and Squashfs ==== | ||
+ | |||
+ | These are read-only filesystems. you make them, and it's a compressed file, but it can be mounted. | ||
+ | |||
+ | Squashfs: pretty standard. used on many linux installer. similar ratio to tar. can use zstandard. not best radio, but deduplicated, | ||
+ | |||
+ | For backup, borg also allows mounting, is deduplicated and compressed. Sqsh is just a "read only image" equivalent with compression. | ||
+ | |||
+ | Dwarfs: found it on github (by mhx?). faster then zpaq, better then squashfs. have not vetted source code though. use at your own risk. it is a cool FS though. | ||
+ | |||
+ | |||
+ | ===== Other random tools ===== | ||
+ | |||
+ | ==== fpart ==== | ||
+ | |||
+ | This is something I didn't know existed until I tried doing it manually. Basically, say you have 5000 photos to back up. | ||
+ | |||
+ | Say you want to send them to older family members with DVDs (yes, I know, optical media in 2021, give me a break). | ||
+ | |||
+ | DVDs can fit 4.4 GiB each, so you want to take advantage of that and split the files evenly. Doing it by hand sucks. | ||
+ | |||
+ | fpart can take the folders and make file lists for several partitions, either by target number of partitions, OR, by space. | ||
+ | |||
+ | eg: to split into 5 partitions with the file lists named list.0, so on | ||
+ | fpart -n 5 -o list -v . | ||
+ | | ||
+ | Problem is, it doesn' | ||
+ | sed 's/^ *//' < list.0 | xargs -d ' | ||
+ | | ||
+ | Where list.0 is the text list, and folder0 is the target. | ||
+ | |||
+ | The --backup=t makes sure that if there are duplicate file names, it will rename one automatically and not overwrite. | ||
+ | |||
+ | NOTE: THIS REMOVES THE DIRECTORY STRUCTURE, AND LUMPS EVERYTHING IN ONE FOLDER. | ||
+ | |||
+ | This is easy enough to then burn onto a CD/DVD with K3B. While you're at it, do yourself a favor and use PAR2 to add parity. | ||
+ | DVDs will get scratched and I currently leave ~5% parity to recover files. |