• he/him

I occasionally write long posts but you should assume I'm talking out of my ass until proved otherwise. I do like writing shit sometimes.  

 

50/50 chance of suit pictures end up here or on the Art Directory account. Good luck.

 

Be 18+ or be gone you kids act fuckin' weird.

 

pfp by wackyanimal


 

I tag all of my posts complaining about stuff #complaining, feel free to muffle that if you'd like a more positive cohost experience.

 


 
Art and suit stuff: @PlumPanAD

 


 
"DMs":
Feel free to message as long as you have something to talk about!


plumpan
@plumpan

Actually kinda bullshit that most 7zip benchmarks run across all cores when realistically most people will be working with LZMA1 files that can't make use of threads to speed up decompression.

Tell your local redump set packager to use LZMA2!

EDIT: LZMA2 alone don't do shit, more testing underway.


plumpan
@plumpan

Size: 8525905920

(8131 MiB)

7z x Method = LZMA:24 real 6m24.943s

7z a -t7z -m0=lzma2 -mx=9 real 3m50.738s (and 17GiB of ram use!)
7z x Method = LZMA2:26 real 2m5.788s

Better, but it's still not really using threads.

pbzip2 real 0m53.717s
pbzip2 -d real 0m19.985s

EDIT: That ain't shit either.

lbzip2 real 0m33.681s
lbzip2 -d real 0m19.202s

Better implementation of the same thing, just way faster compression. Ram use was negligible, none of that 17GiB nonsense. And I saw even closer to 2x compression speed on a 6c12t system.

End Edit

And it goes wide, wonderful! But what about the filesizes?

7Z LZMA:24 7854112521 7490MiB
7Z LZMA2:26 7762319712 7402MiB
BZIP2 8037017095 7664MiB

So I'm saying it right now. It's 2023, romsets are not scene bullshit. There's no reason to cling to inferior methods because that's the way it's always been. If you cared THAT much about saving every single byte, you'd be using LZMA2 already. But you're not. So,

Romsets should use bzip2

And really they should be in tar as well.

Oh, and I guess benchmarks should be in bzip2 instead of 7z.

lbzip2 does the same thing even faster, too


You must log in to comment.

in reply to @plumpan's post:

looks like the fact it's been years since I dealt with compression is coming to bite me. utilities that used libbzip2 had to be updated to support multi-stream bzip2 files and some implementations (like python's bz2 library) still don't support it. it's actually funny because you can still find bugs caused by pbzip2 created files to this day (ie seeing the EOF marker and ending decompression despite the remaining data to decompress)

That's kinda scary that two programs can produce incompatible files like that.

Also just realized that lb was running so fast on my desktop that I was hitting IO limitations, it is in fact twice as fast.

And you are correct, lbzip2 seems to be a better implementation. I'll probably start using that when I do compression things locally.

The main point I want to make in the post is for people to stop using blasted off the shelf 7zip to compress 8GB ISOs because it takes ages to decompress regardless of how beefy your computer is.

there's a bit more happening behind the scenes: most romsets get ran through torrentzip or torrent7z, which takes an existing archive, strips all metadata and repacks it in a deterministic way that hasn't changed for a long time. the people that grab a set once or just grab individual files don't have to care, but it's too helpful for regular maintenance to stop doing