In light of recent circumstances, a bunch of people are now looking to remember the old ways. The days of the Wheel and the Chalice, when snake cults roamed the earth, when you wrote your own engine because you had no choice be needing to go out and remember the old ways, and that's what the Dark Gods demanded, and and and and...
ahem
Okay, so I've made... a few of these, at this point, and ported a bunch to various things. Dredmor used its own engine; CE used its own engine; and even before those two things were kicking around I did a bunch of stuff including Trying To Fix, But Failing To Fix, The Goddamned Lithtech Engine. I've also used a lot of commercial technology: Unreal, idTech, Lithtech, and even more delightful things like Criterion Renderware.
So here's some nebulous thoughts from the trenches. I don't know if this is actually helpful to anybody, but here goes.
- So before we look at the problem of writing your own engine, what's great about it? What are the advantages and disadvantages? The advantages to my mind are threefold.
First, you have a model of how the entire game works and access to everything. If something goes wrong, you can fix it. If you look at what Brian Bucklew is currently experiencing with the Caves of Qud input system (which he has tweeted about repeatedly on the Bad Site). As a CTO or technical director, one of your jobs is to de-risk; to my mind, a key element of de-risking is making sure you have the source code to whatever you're using so you can actually fix stuff if there are problems.
Second, you can really customize things to your needs, including your workflow. This has the potential to improve your iteration time once you really get going, which I think is the most important thing in indie game development (or any sort of software programming, to be honest.)
Last, a technology base you make for one game can be moved to other games. To quote CEO Nwabudike Morgan, "One does not simply take sand from the beach and produce a Dataprobe." Back in my undergraduate days, I paid my way through university by doing contract work. One of the more interesting contracts to come my way was working for Derek Smart (yes, that Derek Smart.) One of the interesting things about his oeuvre is that it's a series of successive refinements towards an ideal space game which he has in his head, and which has a severely loyal follower group that is willing to buy a surprising amount of space simulator and who were quite happy to pay for successive refinements towards this vision.
When I got into the codebase, I was hired initially to modernize the engine and move it to Direct3D 9 for a new space game ("Galactic Command: Echo Squad"). (Inside the codebase were things like wrappers that reimplemented bits of Fastgraph, a DOS/Windows/Windows CE graphics library originally written sometime in the 90s by Ted and Diana Gruber. Their web page is still up.) I was scornful, and it was AWFUL to modernize and some things that were upgraded I don't think either one of us were happy with, but many years later I realized the genius here. At any point, starting a new project, Derek could go back to the previous game, and plan a series of incremental improvements to the next one - and always had a starting point. This is incredibly valuable.
One of the big struggles I have in doing small game projects as a sideline to my current job is that I don't have a starting point, and I am at ground zero with a lot of experience. That makes it very hard to get enough inertia to keep going. By virtue of having a starting point, the process of getting to the next finishing point is that much easier. Jeff Vogel at Spiderweb Software does pretty much the same thing. These are people who have managed to stay in business for a very, very long time.
The disadvantages of course are that you don't have a base of technology to start from. You're paying for a grab bag of tools which are theoretically battle-tested, and which may or may not be suitable for your game, but by gosh there's a big pile of them. A lot of the things in an engine are things you may not want to implement yourself or may be bad at implementing yourself. I think a lot of the attraction when licensing an engine is around graphics, and around tooling - and around that idea that you can just grab the thing and start right up. Those are good advantages! It's important to understand the trade-offs here! For now, though, let's assume you're taking the path of the Wheel and Chalice here.
- First, you must understand the graphics part is actually the easy bit. Graphics cards are powerful these days, and while you should avoid doing stupid stuff, you can get away with doing a lot of stupid stuff. (Dredmor's renderer is 100% software - it fills out a frame buffer and dumps it right into SDL 1.2, which internally maps it to... Windows via.. DirectDraw? GDI? Who knows?) There are good libraries out there for more powerful graphics, and you can and should steal them. bgfx is pretty good. Hades, I believe, used Confetti by The Forge.
Are you going to code your engine to the point that you'll have AAA graphics? Nah. You can't compete with Unreal, and even if you could, you couldn't afford to make the graphics anyway. You will do something simple. For that sort of thing, the math is not particularly scary (I say this as somebody with a math degree, and I probably shouldn't be trusted.) Good, really good, resources exist to explain it to you, and I might post some here in a follow-up thread.
The real question is "how do you structure your game logic?"
Dredmor's engine is pretty straightforward under the hood. There is a single thread, and that single thread is responsible both for rendering, handling input, and doing the game logic. There's also a pretty straight forward OOP sort of implementation of everything, because that was kind of okay when I started working on Dredmor's code back in... what, 2005? When you click any object in the world (monster, door, Lutefisk) it goes into the Clicked handler for the object. The cool thing about this is that the clicked handler can, in turn, run whatever animations it wants to run, and can pump the renderer and input loop until it's done doing whatever needs to happen. For Dredmor, which is pretty asynchronous, this turns out to be very nice. For instance, if you hit a monster:
void Monster::PlayHurtAnim()
{
Sprite * oldSprite = bob.curSprite;
if (curPlayer->tileX == tileX+1 && curPlayer->tileY == tileY)
{
PlaySoundFX(parent->hitSound.c_str());
PlayOnBobForTicks(&bob, hitRSprite, tweakVal("monster hurt anim time"));
bob.SetSprite(oldSprite);
}
else if (curPlayer->tileX == tileX-1 && curPlayer->tileY == tileY)
{
PlaySoundFX(parent->hitSound.c_str());
PlayOnBobForTicks(&bob, hitLSprite, tweakVal("monster hurt anim time"));
bob.SetSprite(oldSprite);
}
else if (curPlayer->tileX == tileX && curPlayer->tileY == tileY-1)
{
PlaySoundFX(parent->hitSound.c_str());
PlayOnBobForTicks(&bob, hitUSprite, tweakVal("monster hurt anim time"));
bob.SetSprite(oldSprite);
}
else if (curPlayer->tileX == tileX && curPlayer->tileY == tileY+1)
{
PlaySoundFX(parent->hitSound.c_str());
PlayOnBobForTicks(&bob, hitDSprite, tweakVal("monster hurt anim time"));
bob.SetSprite(oldSprite);
}
}
(In Dredmor land, a 'bob' is anything hanging out in the game world - basically a sprite with rendering logic, but no game logic. the name comes from the Amiga. PlayOnBobForTicks() does exactly what it says on the tin. And yes, this is a bit verbose and could be refactored a bit, but I think a lot of this was written during the Sleepless Nights period of Dredmor when I was seriously bombed out on coffee and pulling all-nighters integrating whatever David drew during the day. Good times, good times.)
(Note that a bunch of this may not be entirely true any more. For instance, somebody might have added some background loading code, and that might have changed things around a bit. Somebody might also have fixed that wretched save corruption bug. But I digress.)
EDIT: It occurs to me that this is basically a coroutine in disguise. :)
In CE, the renderer and the game loop were on different threads. The original thought we had was that the game logic for CE would be complex enough that we would want to multithread it across multiple cores. Consequently, the two threads are asynchronous. Objects internally communicate with each other by a message passing system, which Micah and I wrote and which was based around templates. You could, for instance, define things in C++ like so:
DECLARE_CORE_MESSAGE( NewOfficeJob, gameObjectHandle, gameObjectHandle, std::string, std::string, bool );
DECLARE_CORE_MESSAGE( NewUniqueOfficeJob, gameObjectHandle, gameObjectHandle, std::string, std::string, bool ); // don't post this job if one with the same name already exists.
DECLARE_CORE_MESSAGE( CancelOfficeJob, gameObjectHandle, gameSimJobInstanceHandle );
and objects could either receive messages and respond to them, or not. For instance, here's the gameWorkshopManager:
class gameWorkshopManager
{
public:
std::map<gameObjectHandle, gameBuilding> buildingEntries;
DECLARE_MESSAGE_DOMAIN( simulation );
DECLARE_RECEIVED_MESSAGES_WITH_OVERFLOW;
DECLARE_RECEIVED_MESSAGES(
AddWorkshop,RemoveWorkshop,
AddOffice,RemoveOffice,
AddHouse,RemoveHouse,
AddOfficeStandingOrder,
FillProductionMenu,
FillOfficeMenu,
SetWorkshopJobIcon, SetWorkshopJobQuantity, SetWorkshopJobType,
IncWorkshopJobQuantity, DecWorkshopJobQuantity);
DECLARE_RESPONDED_MESSAGES(NewOfficeJob, NewUniqueOfficeJob, getBuildingModulesGameSide, GetWorkshopAssignment );
void Receive ( const AddWorkshop &msg );
void Receive ( const RemoveWorkshop &msg );
void Receive ( const CompleteJobOrderUnit &msg );
void Receive ( const CancelJobOrder &msg );
void Receive ( const SwitchJobOrderMessage &msg );
coreMessage *Respond ( const NewOfficeJob &msg );
coreMessage *Respond ( const NewUniqueOfficeJob &msg );
coreMessage *Respond ( const GetWorkshopAssignment &msg );
and it sort of gets bulkier from there.
The advantage to this is that messages could be sent into, and out of, Lua space, and that you could intercept messages at transmission boundaries and put them in the right queue and so on and so forth. It also provided a set of concrete, well-defined boundaries between the scripting world and the game/render worlds, which made it easier at the point at which we just madly decided that everybody should be writing game logic.
The disadvantages to the system were many:
- it is nearly impossible to iterate on fast and/or debug with the level of additional tooling we were able to muster
- messages, being defined on the fly, do not contain named elements and are tuples under the hood, so good luck actually remembering which parameter is the workshop name
- the magic behind DECLARE_RECEIVED_MESSAGES() and in fact the ability to define a message with a macro played merry sodding hell with our compile times.
It was a cool science project, and I continue to think that things sending messages to things really is the right paradigm for game logic - but a lot of this should have been done with a precompiler and not C++ templates, or better yet it should have been entirely in interpreted language space; a lot of tooling work should have gone into it to make this readable and debuggable; and ultimately given that the size of the "engine team" was small indeed, we shouldn't have done this.
(There is actually a long list of things we shouldn't have done in CE which would make for a very interesting article one day, but I think chief amongst them was that we probably shouldn't have made that game. And we shouldn't have used the STL as much as we did either, that is to say "at all". Oh well.) The bigger sin here - the game was never complicated enough to demand any of this! Keeping the rendering thread and the game thread separate is a useful and good thing, but our game logic was never going to be slow enough that we would be bottlenecked by it - we were mainly bottlenecked by pathfinding and trying to make decisions about what logic to trigger next for the AI. That would have benefited tremendously from not trying to be clever, writing it in C, and just leaving well enough alone.
So. If you're going to write code for a game, it doesn't have to be an engine, and you should probably structure your logic to be as simple as possible for the game you want to make. I actually think that's one mistake being made by a lot of people while we all navigate the discourse - the notion that if you aren't going to use Unity/Unreal/Godot/whatever, you have to Write Your Own Engine. Dredmor really isn't an engine per se; it's very deliberately minimal in what it sets out to do, and I think other than a few eccentric things (such as the font handling, which is eighty-six functions copy pasted to reference different fonts with names such as PrintTextAt_G_SF_R()) it worked quite well at doing what it did. CE was a much more general framework - one can imagine making a lot of games based around that engine that are not that game, and it included a bunch of really neat things like an entire retained mode UI toolkit (which we also shouldn't have used) - but ultimately, I don't think it did a good job at helping us make a successful game. (We wrote a lot about CE on the Gaslamp blog, which is still up as a monument to the folly of dwarvenkind.) It is possible to just Write A Game, and I kind of recommend this.
That said, holy shit I learned a lot from writing CE.
- You may have noticed I never answered the question of "how do you structure game logic?" The point is that I don't actually have a great answer, because I think it depends on your game - which is one of the nice things about writing your own game from scratch. If you know what you're trying to create, you can tailor your organization to suit.
I don't know how to do serialization either (the act of loading or saving your game state to disk). Dredmor does it in a very binary fashion.
Dirt::Dirt ( FILE *fp ) : GameObj(fp)
{
planted = (ReadSi32(fp) != 0);
dirtCounter = ReadSi32(fp);
Build();
}
bool Dirt::Save ( FILE *fp )
{
bool okay = true;
okay &= GameObj::Save(fp);
okay &= WriteSi32(fp, planted != FALSE);
okay &= WriteSi32(fp, dirtCounter);
return okay;
}
Yeah, right from a Unix file descriptor, baby. CE does a more sophisticated thing:
void gameBuilding::typeSerialize( typeSerializationNode* node ) const
{
node->serializeChild("currentWorkshopAssignment", currentWorkshopAssignment);
node->serializeChild("currentEntry", currentEntry);
node->serializeChild("numInteriorSquares", numInteriorSquares);
node->serializeChild("name", buildingName);
node->serializeChild("buildingObject", buildingObject);
node->serializeChild("workshopModules", workshopModules);
node->serializeChild("nextWorkshopID", nextWorkshopJobID);
node->serializeChild("office", office);
node->serializeChild("officeJobs", officeJobs);
node->serializeChild("officeStandingOrders", officeStandingOrders);
node->serializeChild("workshopModuleList", moduleList);
node->serializeChild("upgradeLevel", upgradeLevel);
node->serializeChild("upgradeRequirementsMet", upgradeRequirementsMet);
}
void gameBuilding::typeDeserialize( typeSerializationNode* node )
{
node->deserializeChildOnTo("currentWorkshopAssignment", currentWorkshopAssignment);
node->deserializeChildOnTo("currentEntry", currentEntry);
node->deserializeChildOnTo("name", buildingName);
node->deserializeChildOnTo("buildingObject", buildingObject);
node->deserializeChildOnTo("workshopModules", workshopModules);
node->deserializeChildOnTo("nextWorkshopID", nextWorkshopJobID);
node->deserializeChildOnTo("office", office);
node->deserializeChildOnTo("officeJobs", officeJobs);
node->deserializeChildOnTo("officeStandingOrders", officeStandingOrders);
node->deserializeChildOnTo("workshopModuleList", moduleList);
node->deserializeChildOnTo("upgradeLevel", upgradeLevel);
node->deserializeChildOnTo("upgradeRequirementsMet", upgradeRequirementsMet);
node->deserializeChildOnTo("numInteriorSquares", numInteriorSquares);
}
There is a significant block of internal logic in CE that does a giant messy search to fix up pointers to things afterwards. This caused a lot of problems, because (whyyyyyyy?) CE runs as a 32-bit application. Combined with the fact that you can have lots of objects and handles and who knows what running around, you can blow your memory to the point where we actually blew up the 32-bit address space.
I do believe you should store saves in an intermediate file format that is not binary. Dredmor unequivocally got this wrong, and I apologize. Use XML or JSON or some other such a thing. CE used XML and stuck it in a ZIP file. Probably the right way to go about it. (I don't have a solution for doing the pointer fix-up dance, but gosh it is convenient to have it. I suspect there's probably a really clever way to do this that I just don't know about.)
- There is a school of thought that insists that you have to write everything using specific paradigms, that you should be using data-oriented design or entity component systems, or writing it all in C, or whatever. Yes, but also no? For ninety percent of indie games, you are not going to be bottlenecked by rendering. You're probably going to be bottlenecked by whatever stupid stuff you're doing in your game logic, and it's going to be dumb, and you'll profile it and fix it. Instead, your goal in life is to make it from the beginning of writing the game to the end of the game without falling into Hideous Traps.
Here's a non-exhaustive list of Hideous Traps:
-
Memory management. C++ is not a garbage collected language. It is very easy to leak a lot of memory all over the place and not know how or why. In their infinite wisdom, the C++ standards committee provides us with things like std::unique_ptr<> which I frankly do not like, because they pretend memory management is this weird voodoo thing that can't possibly be reasoned about. I prefer to actually allocate and deallocate memory explicitly, and to be able to have a mental model that lets me reason about it. The data-oriented design people are explicitly right about the fact that your code does not do arbitrary things - there are a set number of things that happen in video games, and you should know what those are, be able to reason about them, and understand your memory life cycles accordingly. Allocating memory from a big array is often a really good idea. (And if that doesn't work? Well, we have ig-memtrace.) I also find this makes it impossible to debug your code, which brings us to our next point:
-
Not being able to debug your own code. This happens more often than you think: somebody engineers enough layers of complexity into their codebase that the debugger no longer functions correctly. STL makes this a very real, frightening possibility. Anything templated will usually break the Visual Studio watch window like a tiny, dying twig. (This was a huge, huge pain in the ass for CE.)
There WILL come a point where you have to use a debugger on your game. This can fail in one of two ways: first, your code is too slow to debug in debug mode, and cannot be debugged in release mode with symbols on because C++ is inlining something. Second, the debugger just spazzes out because you're five layers deep in STL template voodoo and suddenly you're stepping into weird messes of private members trying to find the actual object causing the trouble. Then you're into logging and printf() time and nothing good happens here. The debugger makes you faster! Use it. Plan to use it.
-
The third hideous trap is "not architecting for a really important feature when it's easy." Here, some amount of planning is good. The classical example is the undo function. You can tell which tools planned for undo/redo from day one, and which ones did not, and it's the ones where undo/redo don't work. (I seem to recall the old Torque Game Engine was pretty bad for this.) In games, your problem is loading and saving. See discussion above. Be aware of what will come down the pipeline, and plan accordingly. If you are writing an editor and want undo/redo, literally the first thing you should implement is the command pattern that makes this possible.
-
The fourth hideous trap is probably string handling and localization. I've never gotten this right.
-
The omni-trap, above all traps, and which CE ultimately fell into, is this: do not build things in such a way that you cannot iterate incredibly rapidly. This is the SINGLE MOST IMPORTANT THING IN WRITING A GAME FROM SCRATCH. Make sure you can change stuff fast, compile fast, debug fast, link fast, and load your game fast. The accumulated cost of not being able to do so will hurt you, repeatedly: you need to be able to get into that play-test-design loop, and stay in that play-test-design loop. Most tools are really bad at this!
- You have a hell of a lot more resources than Gaslamp ever did. Some of my favourites:
- the powerful Sean Barrett has this giant collection of single header libraries. You should use his image loader. I would also use his TTF reader. You should use ... basically, a lot of it, to be honest. (Incidentally, this is the little piece of code he wrote for me when I was completely stuck on how to make fast updates to connected components in CE work, and by god it was a marvel to watch him do it.)
- SDL exists and is great. Use it. It makes your games portable. With the new 3D toolchain that Ryan's working on, this is a pretty viable path moving forward for lightweight 3D titles as well.
- Dear IMGUI basically gives you an easy to use graphical user interface for all your tooling, and I do mean all your tooling. Oh my gosh, I wish we'd had this back in the day.
- I really like this small collection of C array algorithms. If you combine this with Sean's dynamic array and hash structure stuff, you basically get 98% of the STL, without the suck. You still need a good string library, and I'm working on finding one I actually like.
- A nice collection of lock-free data structures.
- This is the best profiler.
- This is what you use when you have memory leaks.
In general, look for libraries that exist as single headers, or header + source file, and with minimal dependencies. This is a sign that they are written by a master programmer who knows their shit.
(You will find that you may need to intersect a polygon with another polygon. This will happen to you. You want to use Clipper You may need to triangulate a polygon. This one is harder! These days you should use Earcut or maybe Marco Livesu's segment insertion based earcut If you require a Delaunay triangulation, which is to say a triangulation where the triangles are provably "nice" in a way I won't go in here, I'm writing my own and will try to get it out this year. Floating point robustness is really hard. This was another problem in CE - the lack of a good triangulator. I tried to adapt one instead of going deep and writing my own, and this was a bad idea. A few years ago, I wrote my own, and suddenly things stopped exploding any more. Who knew? Floating point is hard.)
I think the biggest single problem in indie game development these days is the lack of good level editors, and maybe this whole mess is the thing that makes me get off my posterior and write one. But the engine and game itself is pretty simple and not something to be afraid of.
Next post: so what am I doing for my hypothetical game?
