scripts:backups_and_data

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
scripts:backups_and_data [2022-04-26 07:43] tonyscripts:backups_and_data [2022-06-03 20:42] (current) technotony
Line 6: Line 6:
 ==== Borg ==== ==== Borg ====
  
-I use Borg for my current active backup on my laptop. Since the laptop is only schoolwork, there isn't a ton of other files to copy. +[[scripts:backups:borg|Borg]]
- +
-Borg has a few cool features: +
-  * Incremental backups only copy the differences +
-  * Ability to mount the backups easily for recovery +
- +
-For my desktop, I use good old tar files generally for system setup. All important data on this system is generally copied to one of many external drives. +
- +
-I used to use Syncthing, to sync files across my systems. I later switched to unison over ssh through my own VPN. I now just use rsync for simplicity and speed. +
- +
-I have various drives, some old, some new holding backups. Critical things like family documents are redundant on my drive, at least 2 externals, and an encrypted copy on Backblaze B2. +
- +
-Old drives with bad partitions are generally retired and become permanent backups after I load an important collection and leave it. I try to power them up twice a year to ensure they still work. +
- +
-I use PAR2 to create parity files. I do this for both TAR and ZPAQ. I have 2 optionsuse files/folders and if bitrot happens, only lose 1 file, or use archives/containers, where bitrot can kill everything. However, using parity with 5% prevention should be enough to protect the entire archive. +
- +
-If I have to encrypt anything, I use AES256 with GPG. If there is a fault in AES, lets be real, everyone will be screwed anyways.+
  
 ===== Parity ===== ===== Parity =====
  
-There are a few option. Par2 (parchive) seems to be the standard here. Of course ZFS and BTRFS can keep track of parity bits in RAID setups, and ZFS will automatically correct the bit errors from the parity drive (as long as you don't pull an LTT and periodically scrub the drives). +[[scripts:backups:parity_par2|Parity and Par2]]
- +
-There is also parity mode with RAR and Dvdisaster, though I have limited experience with both. Par2 is kind-of inefficient with the overhead per file, and it was traditionally run on a single tar or zip, etc, not multiple files. Par2 supports multiple files (inefficiently). I still rather par2 a SquashFS or ZPAQ file with the space savings of those plus single file mode is more efficient. +
- +
-==== Par2 larger files ==== +
- +
-When trying the multi file archive, I was annoyed that it would add the Digikam XMP sidecar files (used to avoid writing EXIF into videos and RAW photos), which is a waste of space. +
- +
-So, instead of giving a path, you can give a list of filenames and filter the larger files (or by name like *.ARW or *.NEF). +
- +
-  par2create -v -m4096 -b32768 -c1023 -t24 -R rec -- (find -type f -size +200k) +
-   +
-This is intended for fish shell, hence no $. This uses a max of 4G of memory, uses the maximum allowed 32768 blocks (meaning max 32k files), with 1023 parity blocks (allowing us to lose 1023 of the 32768 original blocks and still being able to recover). -t24 uses 24 threads, change this for your system. rec is the filename and -- marks subsequent arguments as file names (put a path here instead to just do every file). +
- +
-When verifying, I also add -T1 to only do 1 file at a time. This is to avoid thrashing on my spinning rust hard drives. Might not matter on SSDs. It's especially important on optical media, where seeks are expensive, and the media is best treated as a linear archive like tape. (Yes it's random access but that laser carriage is slow vs a HDD head and it's a linear spiral groove anyway). +
- +
- +
  
 ===== Compression ===== ===== Compression =====
  • scripts/backups_and_data.1650959015.txt.gz
  • Last modified: 2022-04-26 07:43
  • by tony