Quelklef

milk consumer

girly but not a girl

name-color: #C09


say, "just hypothetically", you wanted to write some code which had state. and say you wanted to treat the entire program state as a monolith, ie one single object! but say that the state was really big, so it wouldn't be reasonable to just load it all in to memory at the same time. and say that--luckily--most of the time the program didn't need all of the state at once


here's what im thinking as potential solution

you represent the state as some JSON-like data structure which has maps and arrays and anything else you desire. that structure itself has an underlying representation using only string-keyed maps. these underlying maps have two possible representations on disk:

  1. serialized as text/binary file, ala JSON
  2. reified as a directory whose entries' names are their keys in the map. entries are themselves reified values (ie, either files or subdirectories)

hence to access the value programState.key.subKey.otherSubKey.finalKey the program traverses through directories (with names matching the keys) until it hits a file, loads that file into memory, and continues traversing. likewise for writing values

point being the entire program state is able to be treated monolithically, but you're only ever actually loading small amounts of it into memory at once


You must log in to comment.

in reply to @Quelklef's post:

ya the keys would need to be stored in a trie or something

that would work until the trie itself gets too big to be reasonably stored in memory. which would not be hard, just add some huge array to the store

also kv-store doesn't allow for heterogeneity, ie a map where some values are flattened and some are atomic

probably would be a lot faster than disc tho!