es

@ess

she/her

it's me! es. i'm also on bluesky, twitter, and mastodon.

es @ess9/12/2023, 3:12 PM

⚡ Twilight Sparkle @twilight-sparkle

Una @unascribed9/12/2023, 8:25 AM

To Audio & Back Again - Audio codecs as image codecs

If you're anything like me, you've wondered what would happen if you used a compression algorithm for something it was not designed for. A lot of the time, the results are boring — you just get really large output.

However, image and audio codecs are special. Nowadays, psychoacoustic and psychovisual codecs are used — that is, codecs that are tuned specifically for human perception, and throw away information that humans will not notice. A great example of this is that most audio codecs will simply discard all audio frequencies above 20kHz. But the characteristics of the visual system and the auditory system are very different.

To explore this idea, I've created a dedicated page on my site with an interactive demo showing a sample image converted to audio, compressed with a lossy codec, and then back again, with various different codecs and parameters to choose from. That page has an extended introduction, I'll reproduce some of it here:

#The Cohost Global Feed

⚡ Twilight Sparkle @twilight-sparkle9/12/2023, 2:33 PM

This page's posts are visible only to users who are logged in.

You must log in to comment.

in reply to @unascribed's post:

@ROU9/12/2023, 11:53 AM

this is super cool. i've been doing audio databending on images for a while now but somehow i've never thought to try using a compressed format as part of the process.

Una @unascribed9/12/2023, 6:01 PM

I've been aware of Audacity databending for a while — it's interesting the kinds of things audio processing filters do when applied to general data. As a codec nerd, my immediate thought is to use a lossy codec in addition. :P

bark. bark bark @bark9/12/2023, 3:28 PM

ooo this is fun. have you considered adding some voice codecs like codec2 that are specifically designed to squish voice down to tiny tiny bitrates?

Una @unascribed9/12/2023, 6:00 PM

I probably should, but I was focused on "music-ready" codecs here. I feel like a voice codec at voice bandwidths is just going to be a lot of noise.

Una @unascribed9/12/2023, 8:24 PM

I'm working on a revision to the site with more pixel formats and codecs — the results from low-bitrate Speex are actually pretty interesting:

~~FFmpeg doesn't support codec2, so it's out of scope for now as I can't plug it into my encoding harness.~~

bark. bark bark @bark9/12/2023, 9:47 PM

oo yeah i see what you mean about voice codecs. good luck with the revision :3

nortti OH1CAU @nortti9/13/2023, 9:46 PM

dunno about your setup, but I can get codec2 support in ffmpeg on debian through libcodec2

Una @unascribed9/13/2023, 11:50 PM

Someone mentioned this on fedi as well, but I forgot to update the comment here — my specific system I use to get an FFmpeg build doesn't support codec2 unless you turn on an option that builds almost every feature, which I don't want, so, a bit of a rock/hard-place as I also wouldn't want to build FFmpeg manually all the time.

The person that mentioned this on fedi did some test encodes using my script snippet, and found that even at max bitrates codec2 emits completely unrecognizable smears very similar to low-bitrate Speex.

0x10f @0x10f9/12/2023, 3:33 PM

I tried compressing text data as JPEG and MP3 once. IIRC, you could make out some words in the JPEG-compressed text but the MP3-compressed one was totally garbled. Your approach is more interesting.

Una @unascribed9/12/2023, 6:00 PM

Yeah, text is very unforgiving — images and audio can both take a lot of corruption before becoming unrecognizable, while with text it only takes a few bits.

the cat ever @twi9/12/2023, 10:04 PM

takes me back to fucking around with images in audacity a whole decade ago, a bunch of people i knew online were doing that and it was a great time, i still have some of those saved as wallpapers. i've wanted to mess around with it again but been short of novel ideas, so this could certainly be something to play with

lea @lea9/13/2023, 6:46 AM

Maybe I missed it but what order are the pixels in? I was thinking whether something like a Hilbert curve where a following pixel is always a neighbor of the previous one* would change anything. This is probably less straightforward to implement with planar formats and chroma subsampling, though.

*edit: while still true, locality is preserved only in 1D->2D conversion, not the other way, makes sense ig

Una @unascribed9/13/2023, 4:20 PM

I list the encode order in the dropdown for pixel formats — it's the same as the image data, since no transformation is being done, it's just the raw image stream as audio.

Sunfire, in theory @sunfire9/14/2023, 12:05 PM

neat! i wonder what some of these would look like in motion with a video input.
how fast is the conversion back and forth?
i would imagine probably not fast enough to do in real time

Una @unascribed9/14/2023, 7:12 PM

It's… not quick. Here's a video I made with a similar process when I was first messing with this years ago: https://pool.jortage.com/sleepingtown/media_attachments/files/003/168/946/original/241b2158c3f65844.mp4