tef

sometimes you need to write in another language, but you don't really want to. sql is just one of many examples. programmers like being able to write table.Join(table, func).Select(columns), rather than learn the arcane rituals of 1970s era IBM mainframe users, and who can blame them.

another example, albeit less common, is writing a grammar

no-one wants to learn yet another enbf/regex hybrid language, one that doesn't seem to have good stack traces, error messages, and not many people enjoy the pain and suffering of integrating codegen into their build pipeline. that's why they commit python crimes, instead.

(the rest of this post is about python, go, and a bit of ruby.)

when it comes to writing another language inside python, some people go for a functional combinator style approach using a set of functions to build up larger structures:

expr = Rule() # forward definition is a pain in the ass
expr.Define(Sequence(expr, Literal("+"), expr))

other people go for a classical method builder approach.

expr = Rule().Call("expr").Literal("+").Expr("+")

and some people go absolutely hog wild with method overloading.

g = Grammar()
g.Expr = g.Expr + "+" + g.Expr

each approach has some merits but every approach shares the same problem: writing nested structures sucks ass, and it becomes difficult to work out what the code does once things stop being trivial.

expr = Sequence(Capture(left), Repeat(0,1, Choice(foo)))

expr = Rule.Capture(left).Then(Rule().Choice(foo).Repeat(0,1))

g.Expr = g.Capture(left) + g.Repeat(0,1, g.Choice(foo))

this code is so unpleasant to write that one python library abandoned these style of builders entirely. they just built a preprocessor that works at runtime, by using reflection to read the source code, and using the ast library to reparse it.

other people solve this problem by not using python. in ruby, code crimes are normal and regular. pick any library in ruby and you'll find an embedded dsl in it. ruby makes the above examples super easy to write.

g = Grammar.new {
	define_rule "expr" {
		capture { g.Expr }
		repeat {
			choice "foo"
		}
	}
}

you just can't get the same thing in python. there's no blocks, there's no inline functions, but there is a with statement.

it might look a little clunky, but it does work:

class Grammar:
    @rule()
    def expr(self):
	    with self.capture():
	        self.expr()
        with self.Repeat("+"):
	        self.Choice("foo")

this is what i like to call a structural builder, or an object that gets called multiple times, with nested scope, in order to produce another object. it's different from a combinator or method chains because there's no explicit threading of the builder functions themselves.

it's the sort of design pattern that a functional weenie could never bear, but it does make for some nicer code in more procedural languages. here's what one looks like in go:

parser = BuildParser(func(g *G) {
	g.Define("expr", func() {
		g.String("x")
		g.Optional(func() {
			g.Call("expr")
		})
	})
})

again, it isn't as nice as the ruby code, but there are some advantages to this style of builder.

• you don't need to use much deep magic to make it work
• it plays nicely with method chaining. you can have postfix operations like g.String().Optional()
• you can use variables as part of your builder. x := g.Backref(); ...; g.Match(x)
• you can generate great error messages. as each invocation is on it's own line.

even go lets you pull out "where is the calling function declared" from the stack, and it's really not that difficult to format things to look like regular error messages.

ez_test.go:254: error in Grammar(), 3 in total
ez_test.go:258:expr: error in Cut(), cut must be called directly inside choice, sorry.
ez_test.go:259:expr: error in Byte(), cannot call Byte() in text mode
ez_test.go:260:expr: error in String(), String("\t") contains reserved string "\t"

now the functional weenies will rightly point out that this method has drawbacks. it's so much easier to write invalid code, and you can't lean on the type system so heavily. that's ok though, sometimes we're not embedding something so heavily typed, and we can always run the error checks ourselves. we'll produce better stack traces that way, anyhow.

... and that's it.

i just haven't seen much "structural builder" style code outside of ruby, i only know two other python programmers who commit with crimes, and i haven't seen anything like it in go, but i didn't really look that hard. i'm just real smug about getting nice error messages in my embedded dsl thingy, and i wanted to write it up for later.