I have been using jq for years and I love it. Like regular expression engines it can be a bit cryptic and hard to debug, and isn't always the right tool for the job, but when it is the right tool it's really powerful. But it took me a while to get my head around it, and I know others who have bounced off of it completely. I want to talk a bit about some of the reasons for that, and why imagining that jq operators and functions operate directly on JSON values will lead to confusion.
jq is a command-line tool for transforming JSON. It's useful when you have JSON documents, such as API responses from services like Github, Twitter or Elasticsearch, or diagnostic output from your own programs. It's handy in shell scripts and for quick ad hoc analysis of moderately large quantities of data.
In many popular programming languages, you might have an expression like a+b, where maybe a and b are themselves sub-expressions. You expect that at some point the program will "reach" the point where it needs to evaluate the expression, and at that point each of a and b will evaluated to give a single number (we'll assume this is an arithmetic operation for now) and then they will be added together to get a number. So perhaps a is 5 and b is 2 and the result is 7. We can do this in jq:
$ jq -n '
def a: 5;
def b: 2;
a + b
'
7
All well and good. But if that is our model, what is going on here?
$ jq -n '
def a: 10, 20;
def b: 3, 4;
a + b
'
13
23
14
24
There's more than one model we can use to explain this, but I think the most straightforward is one where expressions work very much like they do in mainstream programming languages – each expression returns a single value and operands/arguments are evaluated before operations/calls. But the values returned are filters, not JSON values.
All filters read in values one at a time, and for each value they read in, they output zero or more outputs. length is a filter – it reads in a value and, assuming it was an array or a string, it spits out the number of elements or characters. . + 5 is a filter - it reads in a value and, assuming it was a number, spits out the number plus 5. 5 is a filter. It reads in any JSON value and spits out the number 5. Every expression in a jq program is a filter, even if it looks like a JSON value, it's actually a filter. A jq program constructs a single filter, and that filter reads in our program's input of zero or more JSON values and spits out its output, also zero or more JSON values. (In the examples above, we passed -n to tell jq to make our input be a single null value.)
Some filters are less straightforward. empty is a filter that outputs nothing. Not null, no values at all. .[] is a filter that reads in an array and outputs the values in the array in sequence. What is this nonsense, I hear you say, what's the difference between an array and a sequence of values? I'm afraid I'm just going to ask you to roll with it for now.
In the example above, a was a filter that returned two values, 10 and then 20, for every value input. b was a filter that returned two values, 3 and then 4, for every value input. So what's going on with that + operation?
Operators and functions combine filters to make new filters. The + operator makes a new filter that first passes a copy of its input to each of its operand filters, then takes the outputs from each of them and combines them by adding. Specifically, it constructs a Cartesian product with everything output by each operand filter, emitting one result for each pairing. In the simple case where each operand emits only a single value, we see the simple behaviour we expect from addition, because the Cartesian product of a pair of length 1 sequences is a single pairing.
Other arithmetic operators work similarly, passing inputs to both sides, taking a Cartesian product and doing arithmetic on each resulting pair. But that isn't the only thing an operator can do. The comma , operator, for example, just emits all the results of the left operand followed by all the results of the right operand. The pipe | operator is the most interesting of the common operators, because it feeds the output of the left operand as input to the right operand, emitting its output. And this gets us to one of the things that is quite unintuitive to grasp coming from other programming languages (at least mainstream ones) – because operators and functions act on filters and not JSON values, they get to control what input those filters receive. It's not always the case that filters receive the same inputs as their parent expressions. If your mental model is of expressions evaluating to JSON values, you are going to find a lot of things confusing and feel like you are surrounded by weird hacks and special cases.
I think this confusion was one element that took me a while to work through and come to a proper understanding of jq. But even with that understood, I struggled in practice with how to tackle problems that needed to dig deep into different parts of a data structure and use those results without "losing my place". And I still sometimes struggle with getting surprised at why a long chain of operations doesn't work and narrowing down where it went wrong. But I will leave those things to discuss another day. I'm going to bed.
