I have the following assumptions:
- When compiling a program, you have a lot of the same values (for example identifier
std) - Thus, in compilers, you want to intern values in order to a) save space b) save computing power
- You want to share the code between the compiler and lsp/ide, because a) you don't want to write everything twice b) you want your language server to be correct (and there is nothing more correct than the compiler)
- In ide/lsp you want to be able to update your data when user writes something and you also want to free memory from unused things
- You don't necessarily want to be able to update any data though, for example dependencies rarely change, so you can have a frozen view of them and recompute everything on updates
The question is: how do you do all of that?
Compiler and lsp/ide views of things are inherently different and you want to use different data structures for them. Compilers want to see everything frozen and never free memory, while lsp/ide want to be able to update some things and free memory in a timely manner.
The obvious part is that you need to abstract your code over the interner(s), either through generics (which is annoying, since everything now needs an I: Interner, but doable) or through dynamic dispatch (maybe slower, hard to do given that you may want different "handle" types).
The non-obvious part is what the hell do you even do in the lsp/ide world? Do you employ garbage-collector-like way of walking the data structures from time to time and deleting unused stuff? But how do you make sure no indexes escape the GC view? And how do you make sure you don't run out of indices/reuse them? Do you maybe use reference counting instead? Would that be significantly slower? How do you make sure dependencies don't get the cost allowing partial updates (you probably can't use different interners, since then you'll have different types)? And those are probably not even all the questions...