Show HN: Hashsync – Deduplicate SSH streams against the receiver's files

On a recent trip, while stuck on spotty hotel Wi-Fi and roaming 5G, I got fed up with how much bandwidth commands like rsync and nix-copy-closure waste.rsync's delta transfer compares each file only with the file of the same name on the destination (or, with --fuzzy, a file in the same directory), so renamed, moved and copied files are sent in full.nix-copy-closure can be worse: a rebuilt package gets a new store path, so the whole package is sent again, even when most of its contents haven't changed.So I created hashsync, which runs a command over SSH and deduplicates the command's output, input, or both against an index of the receiving host's files.As an example, for my last NixOS update, nix-copy-closure would have sent 1.7 GiB with --gzip, but hashsync took it down to 0.7 GiB instead. On some of my backups, I've seen up to 2x-10x less data and time to transfer compared to rsync with compression.Two things make hashsync unique:1. It searches at every byte offset of the stream, whatever its format, for blocks of the indexed files. In effect, it works like rsync's algorithm, but across every indexed file.2. It's a transparent pipe, so it works with rsync, tar, or anything else…