documentation of a significant jq/gojq inconsistency re $-variables in function definitions

[pkoppstein]

jq handles "$-arguments" in function definitions as implicit "as" statements,
whereas gojq does not. This introduces a source of potential inconsistency.

Consider for example:

def prod: .[0] * .[1];
def prodEqual($p): prod == $p;

This works as expected in both jq and gojq.

However, if prodEq were defined as:

def prodEq($prod): prod == $prod;

then we would see the discrepancy, as jq implicitly expands
the definition of prodEq to

def prodEq($prod): prod as $prod | prod == $prod;

Because of shadowing, in jq, prodEq would therefore always returns true!

For example:

[1,1] | prodEq(0) #=> true using jq
[1,1] | prodEq(0) #=> false using gojq

[itchyny]

I know this is a wired behavior of jq, and sometimes causes confusing behavior (jqlang/jq#2039), so fixed in gojq.

[pkoppstein]

@itchyny - I'm a bit confused by your comment. I was asking that the discrepancy (between gojq's non-weird behavior and jq's weird behavior) be documented as a difference, but as far as I can tell, you've closed the issue without updating the README.

On the other hand, if by "fixed in gojq" you mean you intend to change gojq so that it mimics jq's (weird) behavior, I would have expected the closure of this issue to be linked, as you usually do, to the corresponding commit.

Not that it much matters, but I'm fine with the discrepancy, and I'd be fine with gojq tracking jq on this particular discrepancy, but as with all important discrepancies, I'm hoping that it will be clearly documented. Not least because this particular discrepancy bit me (quite badly in that I lost quite a lot of time ...).

Btw, I'm assuming that by "wired" you mean "weird" ....

[itchyny]

Sorry, I thought I've written in README before but was not written. And this behavior looks on the edge of jq bad parts and jq bugs. I'm totally knocked down by the weather right now, so judge later.

[itchyny]

When I implemented the compiler I thought this behavior is a jq bug, but since the issue is labeled as not a bug by nico, unlikely fixed in the future, and fixing in gojq simplifies the compiler a bit. I decided to fix this to match the behavior of jq, but it has some runtime cost (a value argument requires two local variables, also difficulty in optimizing constant arguments).

[pkoppstein]

@itchyny - Thanks for the elaboration. I've added some references to gojq on the jq FAQ and would ask you to let me know if you have any suggestions, though of course you're also free to make changes yourself.

[pkoppstein]

@itchyny - On further reflection, it is apparent that the weird behavior of jq
(as illustrated quite nicely by the prodEq example) is actually a bug,
as I will hopefully prove to your satisfaction in a moment. I've also
come to the conclusion that although there is obviously a case to be
made for mimicking some of jq's flaws, in this particular case a
stronger argument for not doing so can be made.

The evidence that it is really a bug is right there in the manual,
where it is explicit that:

 def addvalue($f): ...;                   #1

is short-hand for:

def addvalue(f): f as $f | map(. + $f);   #2

This was the intent all along (I participated in the discussions
leading up to this feature), but unfortunately, the implementation
failed to adhere to this specification -- the point being that in #2,
the shadowing mechanism that is at the heart of jq is operative,
whereas that was lost in the implementation. QED.

(So the only mystery that remains is why Nico added the "not-a-bug" tag.
I think he either meant:

a) it won't be fixed in version 1.x;

or

b) the intention to achieve the equivalence of #1 and #2 is not a bug.)

Whether it's called a bug or not, jq's current behavior is clearly
undesirable at best, which leads to the question: should gojq
implement the clean/intended/desirable behavior?

My argument in favor of fixing the problem has several facets, but at
its core is the idea that gojq has an extremely useful role to play in
developing jq programs. Let me illustrate with a recent experience of
mine, when I had a program that I was fairly certain was correct, but
for which jq did not produce the expected results. So I spent quite a
lot of time verifying that the program was correct, and then further
time checking that I was not overlooking some well-known jq bug. It
was only when I was able to verify that gojq produced the expected
(correct) result that I was able to track down the source of the
problem.