Once upon a time, I had a client with a big problem. They are a AAA games company using code generation to generate bindings for their user interface between C++ and Lua. So far, so good, except they were using GSL, a really old and no longer actively maintained template engine. This framework uses scripts where every line is output exactly as written, except if it is preceded with a . character. Here's a sample:
</tr>
.for deposit
. year = 1
. accumulated = amount
. while year < years
. accumulated = accumulated * (rate / 100 + 1)
. year = year + 1
. endwhile
<tr><td>$(amount)</td>
<td>$(rate)%</td>
<td>$(years)</td>
<td>$(accumulated)</td>
</tr>
Cute, right? Unfortunately, my client had several issues with this scripting language, and I ultimately built my own code generation solution called Panini to help resolve these.
The problem with every template engine for code generation, and it really doesn't matter which one you prefer, is that they only work well in small samples. In the example above, we generate an HTML table, and the code is easy enough to understand. Even if you're unfamiliar with the language, you can probably see that some variables are being declared, and a while loop calculates an "accumulated" value. However, the logic is intermingled with the data, which only worsens as your template grows. But's that the price you pay for nice templating, and it's mostly fine even when dealing with a single unwieldy template script.
Unfortunately, my client had hundreds of these GSL scripts, which had become an absolute rat's nest. Because the scripts mixed their logic with the output, they were impossible to follow and would often just break, seemingly at random. This made developers scared to touch the scripts, which meant bugs were not getting fixed, and features could no longer be added—a classic downward spiral of tech debt.
Compounding these issues was there was no syntax highlighting for the GSL language. This became my client's first task for me, and I wrote them a syntax highlighting file for the Sublime Text editor. This helped me gain a much deeper understanding (but not appreciation) of the GSL language, surfacing several fundamental issues.
Unwieldy logic
Writing clean code in this scripting language was very difficult because logic could only run on lines separate from the template. For example, my client needed to generate C++ return values that could be optional or not, so either std::optional<std::string> or std::string.
In GSL, this was written as:
. if param.optional ?= 1
std::optional<\
. endif
$(param.type)\
. if param.optional ?= 1
>\
. endif
Even a super simple type change becomes an unreadable mess this way! Note also that this sample doesn't say anything about the indentation level. That's because whitespace was not handled automatically and had to be fixed by hand throughout the template.
It was clear that GSL had to go.
Writing C++ in C++
There are many, many code generation solutions for C++ already. There's Inja, a "template engine for modern C++", there's Grantlee for Qt in particular, and you can probably even get underscore-template-loader to output C++ somehow.
But after evaluating these solutions, I realized they don't tackle the fundamental problem of mixing template logic with data. As your application grows, this will always become a maintenance headache.
And then I had this crazy idea. If my client wants to generate C++ code, why do they use anything but C++ to do that? A C++ source file is just text at the fundamental level; even C++ can handle text files.
I sketched out an idea to write generated code directly to a file stream:
std::ofstream sourceFile;
sourceFile.open("helloworld.cpp", std::ios::out | std::ios::binary);
sourceFile << "#include <iostream>" << std::endl;
sourceFile << std::endl;
sourceFile << "int main(int argc, char** argv)" << std::endl;
sourceFile << "{" << std::endl;
sourceFile << R"( std::cout << "Hello, World!" << std::endl;)" << std::endl;
sourceFile << " return 0;" << std::endl;
sourceFile << "}" << std::endl;
sourceFile.close();
Which will output:
#include <iostream>
int main(int argc, char** argv)
{
std::cout << "Hello, World!" << std::endl;
return 0;
}
And there's something there, right? Yes, it's cumbersome in a different way, but you can see the shape of the code being output in the program. Coming back to that unwieldy GSL sample from before, we could now write that like this:
sourceFile << (param.optional ? "std::optional<" : "");
sourceFile << param.type;
sourceFile << (param.optional ? ">" : "") << std::endl;
That's already much more readable, in my humble opinion. But we can improve it even further if we abstract it with a class:
class DeclareParameterCommand
{
public:
DeclareParameterCommand(const Param& param)
: m_param(param)
{}
friend std::ostream& operator<< (
std::ostream& sourceFile,
const DeclareParameterCommand& instance)
{
sourceFile << (instance.m_param.optional ? "std::optional<" : "");
sourceFile << instance.m_param.type;
sourceFile << (instance.m_param.optional ? ">" : "") << std::endl;
}
private:
Param m_param;
};
sourceFile << DeclareParameterCommand(param) << std::endl;
And this became the genesis for Panini, my code generation framework for C++.
Panini
In Panini, we don't use file streams directly but declare an instance of panini::Writer instead. The library comes with several implementations out of the box, like FileWriter, ConsoleWriter, StringWriter, and CompareWriter. We can configure properties like the type of indentation and newlines we want to output with an instance of panini::WriterConfig.
Since we only want to write the file when the output has changed, we'll use the CompareWriter:
CompareWriterConfig config;
config.chunkNewLine = "\n";
config.chunkIndent = " ";
CompareWriter writer;
Now, instead of writing our whitespace directly, we can use built-in commands to keep track of our indentation level and to output the correct sequence of newline characters:
writer << "#include <iostream>" << NextLine();
writer << NextLine();
writer << "int main(int argc, char** argv)" << NextLine();
writer << "{" << IndentPush() << NextLine();
writer << R"(std::cout << "Hello, World!" << std::endl;)" << NextLine();
writer << "return 0;" << NextLine();
writer << "}" << IndentPop() << NextLine();
But this can be improved even further! Instead of tracking the indentation level ourselves, we can use the Scope() command with a lambda to add our function declaration:
writer << Scope("int main(int argc, char** argv)", [](Writer& writer) {
writer << R"(std::cout << "Hello, World!" << std::endl;)" << NextLine();
writer << "return 0;" << NextLine();
}) << NextLine();
Here's something really interesting: The lambda parameter acts as a scope in our C++ code but also indicates the scope in our generated code! If you squint a little, you can actually see the outline of the code we want to generate.
Debugging
Another issue my client had with GSL was that debugging was practically impossible. You run the GSL compiler with your script, but if something goes wrong, it throws an error and gives up, often without a callstack. So developers had to add random print statements everywhere to catch issues as they occurred.
But once they moved scripts to Panini, they found they could just use the debugging tools from Visual Studio. Since the scripts are now written in C++, they are compiled into a single "codegen" executable. Running this executable will generate your code, and you can add breakpoints to read the program state as the script is being executed.
My client called this small change an absolute game-changer for their productivity. Panini now comes with a DebugWriter, allowing you to step through your code line-by-line on the console.
But does it scale?
Of course, the real test for my library is how this solution scales. How easy is it to add new Command and Writer implementations and maintain them? While I can't speak for my former client, I extensively use Panini in my own game. I have 52 custom command implementations that generate 339 source files in C++, Typescript, ink, and even Visual Studio project files. And all that in under 100 milliseconds.
But Panini is still stuck in a weird place. People criticize the library for not really being a tool for code generation since all it does is write text to streams, which is fair. But the commands are also not really an abstract syntax tree (AST) representation of C++ since Panini is meant to be somewhat language-agnostic. I don't know how best to resolve this, and I'm unsure if I actually want to.
Overall, I'm quite happy with my weird little library, and I'm sharing it in the hopes that you'll find it useful too. 🥪
I recently released Panini 1.4.0, which adds the FeatureFlag command and has some modest quality-of-life improvements. You can read the full documentation here, which is also included in the package on Github.
