It's been an interesting week at The Bad Place, where the level design Illuminati has been fighting again. Somebody posted a bunch of Hammer editing videos, and they're really nice! (This person should hang out with us cool kids on Cohost.) A bunch of people have been nerd-sniped into writing better editors, or at least thinking about writing better editors, including myself! Joe Wintergreen is vomiting up blood! (You should get that looked at, my dude.) Ryan Schmidt is cracking under some sort of weird, seismic pressure! JP actually temporarily unlocked his account on the Bad Place to weigh in! (which I'm really, really, resisting doing.) And we finally have some insight into how a major company views the level design and editing process, which is interesting.
I originally wrote a great big post here about why the technical issues that Ryan brought up in thread didn't hold water, in my opinion, from the perspective of somebody who has a Ph.D and twenty-two papers in geometry processing and artists tools for geometry processing. There's a bunch of confusion going around which is worth clarifying.
- Nobody is saying "we should go back to BSP brushes." I think this is a red herring. What Hammer provides are probably meshes that are internally stored as winged-edge data structures. It's certainly how I would store it. It's not painful to edit, you can easily walk the faces of the winged-edge structure to get polygons out, and you can triangulate the polygons (torturous but not terrible.) If you really decide you want something that you can pass into Nanite, this would be a great time to apply a displacement map and/or a conformal Delaunay triangulation.
1.2. You do NOT need BSPs to perform robust CSG. This was true in the eighties. It is not true now. We have the technology.
1.5. I don't think anybody is saying "quad meshes", either, by the way. I think Hammer can support a face that is larger than four quads. This is certainly something you want if you are doing room geometry. If you have a polygon embedded in a quad mesh and you want to maintain quad flow across it at a later point, there are algorithms one can use (this paper by Mikhail Bessmeltsev solves the problem neatly, by expressing quad edge pairing as a stable matching problem.) Anyhow, we should come up with another name for them as a community. "Quad brushes" or "Winged edge meshes" or something. IDK.
- There is some confusion between "static mesh" and "triangle mesh". To my mind, a static mesh is a mesh that is not deformed. One can generate a triangle mesh from a static mesh: any sort of GPU based asset baking pipeline can do this as a precursor, on top of all the other good things (vertex cache optimization, triangle stripping, Nanite-ing, generating Lumen cards).
I think that getting nice editable mesh primitives into a form that Unreal (or any other engine) can render and eat is a problem of cacheing, and a willingness to accept that you will hit a performance penalty on developer machines when you're not using the cached GPU-baked version. (There is a bunch of other discussion and work that Epic seems to have done trying to make it so that triangle meshes can be edited as non-triangle meshes via means of polygroups, which just seems like a classic case of an artist going "we would like this thing" and a programmer going "ho ho, here's a thing easier for me to write that is useless". I do this too, in fairness.)
At the end of the day, however, here's the sad truth. Despite throwing up red herrings in the form of technical objections, Epic has made the decision that they think what their customers actually want is a "triangle-based" mesh editor editor, rather than a "face-based" level editor. Ryan basically comes out and says as much: a mesh editor is a priority, a level editor is not, because it aligns with two things.
First, enterprise customer workflows. I interpret this as being about "virtual stages" and "virtual architecture" versus AAA game developer flows, which is... okay! That's Epic's call to make! It's an interesting statement, to be sure, about which businesses Epic thinks are valuable going forwards, and it aligns with other stuff coming out of Epic's announcement about just who they are going to be hitting up for licensee money in the future. You don't give a game engine out for free and charge for films if you think your customers are primarily going to be indie, or even AA/AAA, video games. We generally call this space "visualization".
Second, Epic seems to believe that mesh editing aligns with a trend towards kitbashing and photogrammetry, and that this trend is desirable. This makes sense if you own Quixel. I don't think it makes sense if you talk about kitbashing, because what you really need when you are kitbashing in the real world is exactly what a brush-based editor provides: lots of big things you can make out of cardboard that you can stick your kitbashed parts to. You probably want those big pieces of cardboard to have nice, obvious marking points and semantic information that you can snap and align your greebles on, and it turns out if you use a flow-based face workflow and not a triangle workflow, you get a bunch of helpful points that you can easily add as snap-to targets in your level editor. (Again, Ryan is right that triangle meshes don't have this sort of information, but nobody is talking about triangle meshes and nobody should be editing triangle meshes directly unless you're making Quake II! They are a terrible, terrible platform for actual modeling flows for anything! They are great for rendering and certain geometry tasks! People should convert higher level representations to triangle meshes when they are rendering, or performing these tasks!) The points about level designer flow and being able to easily edit shapes are better made by other people, and I will leave them to it.
A big missing bit in this discussion is "what is human architecture", which is actually rectangles or variations of rectangles. That's why quads are a natural primitive; and when designing artists tools, we have considerable research showing that you should always allow artists to work in a medium that they are comfortable with.
If Epic now sees themselves targeting primarily photogrammetry and enterprise-scale workflows, and given that Unity is in no state to articulate clear vision for anything, it seems like there is a clear market for new engines. Godot has had a lot of work sunk into it, and that's cool, but I don't think it's trying to hit a number of design spaces. I would like to believe that we are now in an interesting new golden age of game engines, because I really think it's more fun when there's lots of technology alternatives... but they are a lot of hard work. I'm not sure where efforts should land to build engines, or even tools to build engines, or even what an engine is? I do think it's notable that Brian Bucklew over at Caves of Qud basically moved all his codebase into Unity to get a limited number of useful primitives over rolling his own, and it probably makes sense to supply a framework that just supplies those primitives. It would help if I could identify what those primitives are.
Meanwhile, I think I've been nerd-sniped into writing a level editor, and I'm just trying to figure out if it makes sense to do it in Unreal Engine to prove a point or not.