This is an old revision of the document!

I use Borg for my current active backup on my laptop. Since the laptop is only schoolwork, there isn't a ton of other files to copy.

Borg has a few cool features:

  • Incremental backups only copy the differences
  • Ability to mount the backups easily for recovery

For my desktop, I use good old tar files generally for system setup. All important data on this system is generally copied to one of many external drives.

I used to use Syncthing, to sync files across my systems. I later switched to unison over ssh through my own VPN. I now just use rsync for simplicity and speed.

I have various drives, some old, some new holding backups. Critical things like family documents are redundant on my drive, at least 2 externals, and an encrypted copy on Backblaze B2.

Old drives with bad partitions are generally retired and become permanent backups after I load an important collection and leave it. I try to power them up twice a year to ensure they still work.

I use PAR2 to create parity files. I do this for both TAR and ZPAQ. I have 2 options: use files/folders and if bitrot happens, only lose 1 file, or use archives/containers, where bitrot can kill everything. However, using parity with 5% prevention should be enough to protect the entire archive.

If I have to encrypt anything, I use AES256 with GPG. If there is a fault in AES, lets be real, everyone will be screwed anyways.

I used to use XZ for compressing tar files. As of 2021-01, I have switched to ZStandard since it offers the same or slightly better ratio with -22, with MUCH faster decompression.

I will try to do a larger writeup and comparison in the future.

For compressing large amounts of files for archiving, where I won't need frequent random access, I use zpaq -m4. m5 is better but also takes forever. ZPAQ is particularly good if there are lots of duplicates.

For folders with large duplicates I rarely need, I am testing out dwarfs, since it hits a good ratio with -l7 (which uses zstd -22) and allows random access. Anything above -7 segfaults for me.

If you are low on space and don't care about CPU time, it's a journaled archive. Very good compression ratio.

These are read-only filesystems. you make them, and it's a compressed file, but it can be mounted.

Squashfs: pretty standard. used on many linux installer. similar ratio to tar. can use zstandard. not best radio, but deduplicated, and allows random reads.

Dwarfs: found it on github. faster then zpaq, better then squashfs. have not vetted source code though. use at your own risk. it is a cool device though.

This is something I didn't know existed until I tried doing it manually. Basically, say you have 5000 photos to back up.

Say you want to send them to older family members with DVDs (yes, I know, optical media in 2021, give me a break).

DVDs can fit 4.4 GiB each, so you want to take advantage of that and split the files evenly. Doing it by hand sucks.

fpart can take the folders and make file lists for several partitions, either by target number of partitions, OR, by space.

eg: to split into 5 partitions with the file lists named list.0, so on

fpart -n 5 -o list -v .

Problem is, it doesn't really move the files. If you don't care about folder structure (ie: intended as slideshow or something)

sed 's/^ *//' < list.0 | xargs -d '\n' mv --backup=t -v -t folder0

Where list.0 is the text list, and folder0 is the target.

The –backup=t makes sure that if there are duplicate file names, it will rename one automatically and not overwrite.


This is easy enough to then burn onto a CD/DVD with K3B. While you're at it, do yourself a favor and use PAR2 to add parity.

DVDs will get scratched and I currently leave ~5% parity to recover files.

NOTE: Optical media for large backups doesn't make much sense anymore. Even 35/50/100 GB BluRay discs have the cost per GB of SSDs, let alone hard drives. My cheapest strategy is buy an EasyStore (aka EasyShuck) on Black Friday if it's ~$20/TB. I hate chia miners for ruining this.

Enter your comment:
R​ Y M​ V H
  • scripts/backups_and_data.1646805272.txt.gz
  • Last modified: 2022-03-09 05:54
  • by technotony