walking mirage

just had an idea for a keyboard-chat radio mode for use over bandwidth-constrained radio links that achieves usable text transmission rate for weak signals through grammar compression.

just turn each sentence fragment into a frame with FEC, have a pre-shared dictionary for given languages between clients. either assign more common words to take the fewest possible bits, or go farther and do word pairs. put as much logic in the clients as possible with the assumption that nobody is going to try to decode this on something with 4mhz and 16kb of ram.

build the dictionary by using wideband WebSDRs to analyze every conversation amateur radio operators are engaging in using PSK31 around the world, 24/7, until you have enough data. maybe even build different dictionaries for time of year, time of day, and region.

all sentences must be structured and terminated properly punctuation-wise to fit maximum information in limited bits. words and symbols that aren't in the dictionary get dropped down to UTF-8 with associated reduced speed/decode performance

#ham radio #amateur radio

You must log in to comment.

in reply to @atomicthumbs's post:

wiggler @wiggles9/20/2023, 10:33 PM

2097 ham: yeah the compression is built on early 2020s english so you need to talk in an old timey dialect to get good bandwidth

walking mirage @atomicthumbs9/20/2023, 10:41 PM

i was gonna say something about distributing new dictionaries every so often but i like this idea more.

cat @catball9/20/2023, 10:37 PM

Brotli compression is a little like this, in that it also uses a dictionary of tokens commonly seen on webpages (https://en.wikipedia.org/wiki/Brotli#Algorithm)

The corpus they constructed leans strongly toward english, and the dictionary isn't often updated iirc. Zstd similarly you can train a dictionary ahead of time to make runtime compression much faster if you have predictable data

but yeah, if input text is normalized (like how you mentioned structure enforcement) and users operate them using designated structure and tokens (idk like radio nerds probably love to do anyways), a little finite state transistor with nodes at the end storing the symbol for the token, you could deffo make a really tight little compressor :)

I think for text compression in most irl scenarios, more general compression algorithms win out over fixed-grammar state machines iirc because language is so so variable

bark. bark bark @bark9/22/2023, 1:08 AM

came here to mention this :3 if you go read the dictionary there’s a lot of fun stuff in there, every swear word imaginable plus ancient html constructs

sirocyl @sirocyl10/21/2023, 9:56 AM

how else are they gonna shove the strings xmlcockgang and sonyguysfuckpipe into thousands of open source projects

Brian Danger Hicks @ckape9/21/2023, 5:26 PM

Somewhat related I saw an article in PoC||GTFO that used grammar compression as the basis for an encryption system so that if you gave the wrong key it would decrypt to a different grammatically correct sentence

walking mirage @atomicthumbs9/21/2023, 5:30 PM

damn that's cool

Pinned Tags