Deduplicate Files on BTRFS

BTRFS, ZFS, XFS, and many other file systems allow copy-on-write functionality, allowing multiple files to use the same blocks until one is updated.

These is sometimes called “reflinks”, NOT to be confused with hardlinks and symlinks/softlinks. (I'll make a video on this soon explaining the differences).

While BTRFS doesn't have online deduplication, it has 2 ways of deduplicating:

1. Programs such as rmlint can be used to convert identical files into reflinks, saving space.

sudo rmlint -c sh:link -g /path/to/deduplicate

2. Programs like duperemove can link individual extents within files

sudo duperemove -r -d -v -h --hashfile duperemove /path/to/deduplicate