i have never done a longpost before so I'm gonna just see how that goes and if my writing style is totally incoherent then... im sorry! ive never written about programming in a longform way that wasn't scrappy and meant for like, two other people in discord who knew what the hell i was talking about. i've always had the half-intention to start a blog, like, an old fashion Web Site but i never got around to it because i haaaate web dev. i like designing UIs! but not web dev. fuck that. even though I'm like, okay-ish at it. capable enough. sufficiently knowledgeable. adequately acquainted

anyways, one of my Projects that I've been doing on-and-off for the past few years (not consistently and this is my second take on said project) is writing a JVM in Rust. I'd wanted to do this ever since I learned that the JVM had an open specification that you can just... read. and I also wanted to learn rust when I first heard about it so I just kinda went for it.
i thought it was super cool that there was a new programming language that actually had something going for it. as in, I was familiar with what a pointer was and what "heap" meant but I had never used a language where you actually had the opportunity to deal with that stuff. i hated the idea of C and C++, so when I heard of Rust and heard that it was maybe the best language for WASM period I knew I had to give it a try. keep in mind this was like two to three years ago so i'm not a beginner rust user anymore. (thankfully. it was a little bit painful to start but now i would say im like, okay it?)

long story short, my first attempt at writing a JVM (which was also my first attempt at using rust) sucked and went nowhere. I did get a kinda cool looking bytecode stepper

out of it but thats about it. no gc, most instructions unimplemented, and it was definitely riddled with UB. also, the classfile parsing logic was done entirely manually and it sucked and there was sometimes unwrap and sometimes the ? operator (what even is that called? "try"?)

okay but like. i still haven't actually linked the specification so here you go
i don't expect you to read it, but it's slightly more easy to follow than i initially expected it to be. this is what I'm talking about. i assume you know what a jvm at least is, if not then you can read up on it here

anyways, my "second take" on the jvm I started like, a month ago? i think.

classfile parsing

one of my main goals was to make class parsing significantly more concise so I decided to write my own proc-macro to handle creating derives for a trait that would signify not-quite-PoD (plain ol' data, aka, safely transmutable data thats just bytes where any configuration as valid. think casting an integer to a float. or an array of one to an array of the other). I named the trait JParse and the actual usage looks like this:

all types in that struct implement JParse, and that prefix attribute is one of the reasons that the derive macro is so convenient. the way classfiles represent variable length arrays is with a prefix of itself a variable length to denote how many elements are present in the following array. when I say the prefix is of variable length, I mean that it changes depending what data you're trying to parse. usually it's a two byte prefix, so a u16 (big endian) but sometimes the data can be long and uses a u32 (also big endian, because the jvm is big endian) to tell the parser how long the following data is. by default it's like I have #[prefix = 2] for every Vec<T> where T: JParse, and using constant generics the actual change for How Much Prefix To Load is propagated. and it actually works very well! which was surprising. i just kinda wrote all the structs as you would in your head, and I just tagged it with the JParse macro and it just entirely worked. some things I had to manually implement but that's due to it having some weird logic that would be entirely too annoying to accommodate in a proc macro meant for like... 40 structs total that are almost exactly the same in generic make-up. but after all that I'm left with some structs with some numbers and blah blah blah I won't bore you with the details but a lot of the parsed structs reference other structs using indices into Some Parent Struct Or Other Struct That Is Mentioned In The Specification meaning that as is, the data is not very usable in a live environment with java stuff being thrown around. like, you don't want to have to resolve 100 things and do 100 type checks if you want to load a single variable off the stack, so I instead "resolve" all of these raw JParse'd structs into more convenient structs where things like strings are no longer just indices into the "constant pool" but they are just like, Arcs, for example.
(the constant pool is a thing that each java classfile defines. e.g. when you are dealing with the bytecode and it wants to load a string that the Java User wrote, it doesn't literally put that entire string of UTF-8¹ bytes but an index into a list of "constants" and you resolve said constant to find out what string you want to put onto the stack.)

preliminary wasm experiments

so, WebAssembly is a thing that runs in your browser. it's like Essence of JavaScript in the sense that it's syntax (it is a bytecode and yet it has a syntax, kinda) is remarkably similar to javascript's in the sense that there is structured control flow. like,
if (something) { do this; } else { otherwise do this; }
machine codes do not have this! they have branching instructions which can arbitrarily jump to any memory address, absolutely whenever they feel like it.²
and java bytecode is also the same! it has branching instructions that simply specify an offset, as in, how much to either backtrack or go forward (in bytes) to get to where the bytecode wants to go. and I don't think the JVM gives many guarantees about what kind of control flow can or cannot appear, meaning that I have to deal with this somehow and turn unstructured control flow into a bunch of if statements and do while loops.
you can read about how you do that here because honestly I only barely understand it and I'm like 80% confident I've fully implemented it correctly.
i don't have a lot to show for this because... it's bytecode. like, i can give you a screenshot of the stdout when I debugged stuff to read the output, but seeing java bytecode and some magic hand-wavey looking output with some control flow added (especially if you cant read WAT, or the webassembly text format) honestly doesn't sound very impressive, and due to the nature of debugging, I don't have those debug statements in place anymore so I would have to go back and re-do that and egh too much effort. just trust me that it wasn't super easy. I think I spent like over twenty hours on this part? and this doesn't even implement anything else. it's literally just control flow. there's no garbage collection, no heap allocation or anything. it literally just converted very simple control flow (and a few actual instructions to be fair) into webassembly. I got a basic fibonacci program working, along with LCM, so that was cool. but... yeah. not a whole lot to show for it. maybe I'll have a web demo at some point.
(if you are legitimately interested, like three posts back was me celebrating control flow working, and there was a screenshot or two)

heap representation

i wrote the section below this one before i wrote this one because i kinda forgot about this one. but i took a dive into rust DSTs (dynamically sized types) which are a little bit of a weird feature. cool, but kind of half-assed. even the offical rust nomicon says so.

i thought it would be cool to have as rusty as a wrapper as possible for the raw data types stored on the java heap (for when you know what kind of object you're dealing with). this is especially useful for types like String which have a specific layout and are also "special" in the JVM as they're used internally.
I came up with this:

T is some type, which is not necessarily Sized, which means that by default RawObject is also ?Sized. This means you cannot store it on the Rust stack, as the compiler generally has no clue how big the data is. you can only deal with it in a pointer or a reference, à la *mut RawObject<T> or &mut RawObject<T>

arrays internally are represented as

(i'm not sure the padding field needs to be there. i'm rather certain it's doing nothing and not affecting the alignment of [T] but who knows. okay well someone knows it's just not me!)

where ObjectInternal is an unsafe trait with no methods that simply marks data types which can be stored in an array. the only types which are stored in an array are pointers (not fat pointers, just usizes) or primitive types including but not limited to char, int, or double. to represent Objects, there's another struct called Object which is just a wrapper around a normal pointer (not a fat pointer). that struct implements ObjectInternal, making it very convenient to represent a String in Rust. A reference to a String object on the heap essentially becomes *mut RawObject<Object>. A reference to a String[] object looks like
*mut RawObject<RawArray<Object>>.

(An Object of some type that the JVM is not sure of at compile time is just *mut RawObject<()>, for posterity.)

executing the bytecode

OKAY so finally something that has more substance. coincidentally, "finally something that has more substance" is how I felt the entire time I was writing the jvm up until I began on making the interpreter, because the rest (minus the WASM thing, that has no bearing on the rest of the project atm) was literally just boilerplate.

so the jvm is a stack and register machine, meaning you have a stack of "operands", or, data that you are immediately working with, and some registers, i.e. a place to store local variables. there are also static fields as a place to store data.
the jvm works in "frames", which is the context of execution for a given method invocation. as the jvm starts executing the main method in the main class (the first thing it does, generally) it pops a frame onto the stack which contains the local variables, register, and program counter. the frame is mostly opaque to the bytecode, and it can, i assume, be entirely optimized out by the jvm if it so pleases, however I'm not fully sure on that point.

this maps extremely closely to wasm, coincidentally. webassembly is also a stack and register machine, along with a heap and some global variables.

okay, but anyways, actually executing the code properly is more complicated than just initializing a Vec<i64> for the stack and some other data structure for the locals, as you need to deal with...

"class loaders", which, as you may have guessed, load classes. they're also a major security risk if implemented improperly, which is what happened with log4j. log4j decided to by default just parse a specific string as a location to fetch new java classes, from the network, and then run them! that's obviously bad, but classloaders in and of themselves aren't bad. in fact, they're extremely powerful and are a way to do JIT in java, at runtime, as you can generate arbitrary classfiles and pass them to the jvm and say "run it!".
classloaders "own" the classes they instantiate, in a memory model context and in a literal java context. specifically, a class loader and it's corresponding classes can actually be garbage collected by the JVM, assuming there are no more references to said resources floating around. in addition, java classes "belong" to their class loader, meaning, as far as I know (and have experienced, assuming I'm not misremembering) you can have multiple instances of the same classpath loaded, as long as it was two separate class loaders who instantiated the class. i had a super awful to debug issue where I was setting a static field in a class that was loaded by one class loader, and reading said static field from What I Thought Was The Same Class but the class was actually instantiated by a different class loader! meaning the values were not updated and I had to figure out how to make the thread doing the reading be instantiated by the same class loader. not fun

so anyways, it's not too difficult to get a very basic implementation of class loaders in rust but actually pairing said class loaders with their corresponding Java representations is a bit more difficult, because, yes, class loaders are actual java objects which have to be represented in java because its java code that does the class loading. I still haven't implemented that part but it probably isn't too hard?

anyways, once you have class loading setup you can kinda start to implement bytecode. luckily, control flow is much simpler here as you get to just update the program counter instead of having to re-structure the code itself, because wasm doesn't have a concept of a program counter.

the jvm relies on the OpenJDK source classfiles being present, as the core language features such as Strings are very specifically "java/lang/String", meaning you have to source those from somewhere. luckily, openjdk versions before java 9 keep all of them in a jar file that you can just locate. its called rt.jar or something. anyways, i extracted those, and as an additional test, minecraft (server) classfiles, and tried to run it just to see what I would get.
Disclaimer: i have not implemented anything near all of the instructions. maybe like 8% of them are there (there are like 256, some of them are just variants of each other to be fair so maybe like 100 actually Unique instructions). I'm also sure half of the complicated ones that have to do with method invocation which I do have implemented are done so improperly. but, anyways, if you (me) make your (my) half-assed (Not-Really A-)JVM debug all the instructions that it's reading... you get...

a whopping like... 23 instructions! cool. at least it kinda runs?
unfortunately, even if I had more instructions implemented, I would have to implement a massive amount of native code java methods to get anything to actually do anything, which is a hell of a task on it's own. maybe it would be fun if I didn't have to implement literally every other part of the jvm first. but that's how it goes sometimes

there's also more to write about, specifically the internal ABI for method invocations as I was planning on writing a cranelift backend for JIT, but I think I am too lazy to do that

why did i write this

i dont know! i've never written a long form post before, and i'm pretty sure this is interesting to like max 1 other person out there, and i'm concerned it doesn't even interest me anymore and i worked on the damn thing. i have a knack for only starting projects that are like, minimum 4 year projects which are also coincidentally unfinishable as... how do you even finish a jvm? there are pretty much always going to be things to fix, performance gains to achieve, etc. also i would have to play catch up with fucking Oracle and it's 30 years of experience making java stuff. Oracle with a capital O is like a fucking... fortune 10 company. or 20. doesn't really matter at that point

i think i might need something different to write about. or maybe i don't like writing... i might also just be suffering from Coding Burnout recently, so that's fun. i hope that if you read this far that my writing was somewhat passable? would love to hear your thoughts in the comments. from literally anyone.

1: java doesn't use UTF-8 directly but some weird form called MUTF-8 or something which is UTF-8 but like, one or two things are changed. so i just have another library handle that for me and it just kinda works. oh yeah and java/lang/String classes also silly side note, there is another text encoding scheme called WTF-8. it's also kinda not an encoding scheme but more like a phenomenon that occurs when people don't follow UTF-8 entirely correctly.

2: in reality, kernels are generally very sensitive about what memory any given program can see, so you can't literally jump to just any memory address, also there's probably alignment to deal with. memory has to, as far as I know, be explicitly marked "executable" by the kernel before a program can actually run it. meaning if you just have a part of memory, you probably can't just immediately execute it but instead have to very kindly ask the kernel to make a memory page as executable, which it will almost certainly do unless the kernel in question is iOS and aren't literally Safari. no i'm not kidding, iOS apps are not allowed to do JIT compilation.