I finally got around to some extremely basic level and material loading in my custom engine via JSON files. It's super not optimized but I no longer am hard coding any of the scene in my code which is a big step in the right direction.
Also I'm realizing that I've been doing a great job of "hack in just what I need and make it more generic/better in the future". My new metric is that, for any new piece of functionality, if I don't leave it with a few TODOs, odds are it's over engineered for what I need right now. My new favorite is this one:
// TODO: Write some asset management so we get caching and don't leak these resources
This is in the code that loads materials, textures, and meshes because in each case I just new some object onto the heap, never reuse the objects (even if they are the same), and never clean them up. Is the current code production ready? Nope. But did it get me loading up a bunch of stuff from disk really fast without getting distracted by creating some fancy asset system.