maff

welcome to my web page

last.fm listening

 


 

i am just a litel babie……

 


 

how do i centre these dang images
unci fonts now!
maff.scot


my telegram channel where i post the absolute worst shit you've ever seen
badposts.t.me/

catball
@catball

you can add the flag --rsyncable when making gzip and zstd archives and they'll rsync way faster and generally less than 1% larger than if you didn't

(it puts little synchronization checkpoints in the file that rsync then doesn't have to calculate on the fly iirc)

e.g. here's some flags i like:

tar -I"zstd -T0 -19 --rsyncable" -cvf stuff.tar.zst file1.txt file2.txt ./directory/to/files/

(note that using --long and --rsyncable at the same time might reduce the benefit to rsync speed since the rolling hash sync points added by zstd are based on the compression window size)

edit: the above example works with gnu tar, but -I means something different with bsd/macos tar. will make another example in a bit


lexyeevee
@lexyeevee

When you synchronize a compressed file between two computers, this option allows rsync to transfer only files that were changed in the archive instead of the entire archive. Normally, after a change is made to any file in the archive, the compression algorithm can generate a new version of the archive that does not match the previous version of the archive. In this case, rsync transfers the entire new version of the archive to the remote computer. With this option, rsync can transfer only the changed files as well as a small amount of metadata that is required to update the archive structure in the area that was changed.



You must log in to comment.

in reply to @catball's post:

in reply to @lexyeevee's post:

It follows perfectly from how rsync works, and this also applies to .zip files naturally because the files within a .zip are compressed independently from one another.

I'd be surprised if the savings loss were even 1%. .tar.gz et al has that as a handwave of why to compress the whole serialized archive all at once, when in reality it's pretty unlikely that the compression window will be all that beneficial when spanning separate files.