Behaviours of various implementations can be checked in the comparison project. Please note that the JSONPath Reference Implementation referenced there is stalled/archived and incomplete.
At the time of writing, online sandboxes for RFC 9535 are available in C# and in Rust.
For more background and comments on the status of the CTS and RI, see this this post (also here in the IETF blog).
Public discussion occurs in the IETF JSONPath Working Group's mailing list (archive).
Today the JSONPath proposed standard, RFC 9535, was published, 17 years to the day after Stefan Gössner wrote his influential blog post JSONPath – XPath for JSON that resulted in some 50 implementations in various languages.
Stefan recently shared his perspective with me:
“After my online publication 'JSONPath – XPath for JSON' in 2007, there was initially a long period of silence. My interests had meanwhile shifted more towards web applications in mechanical engineering when I received requests from Glyn, Tim [Bray] and Carsten [Bormann] to jointly publish an IETF standard. Looking back, I am glad to have joined this Working Group. The new insights into the collaborative processes of the IETF, the discussions and meetings were very stimulating. Surprisingly, different points of view always led to a common consensus and finally to the current IETF standard.”
Tim Bray, one of the chairs of the Working Group, added this:
“Not everyone who stumbles into the IETF process likes it, and I’m glad Glyn wasn’t one of those who bounce off. The process – no voting, consensus calls, reviews by a wider and wider community – feels like the most natural thing in the world to me. I’m happy with the way the document came out. As the standard RFC-series language says, it represents 'the consensus of the IETF'. That’s a powerful statement; the Internet, as specified piecewise in a few thousand RFCs, is one of Homo sapiens’ largest creations. The community owes thanks to the editors and the rest of the Working Group.”
Introduction to JSON and JSONPath
According to its defining RFC 8259, JavaScript Object Notation (JSON) is “a lightweight, text-based, language-independent data interchange format”. It is widely used for storing and transmitting data.
A JSON value is one of the following:
a string, e.g. "red"
a number, e.g. 23, 8.95, 3e8, 1e-10
one of the values true, false, or null
an ordered collection of values, known as an array, e.g. [2, "cat", 4]
an unordered collection of name/value pairs, known as an object, e.g. {"m": 10, "n": false}.
A value can be thought of as a tree of nodes, each of which is a value.
Consider a simple example of a JSON value representing the contents of a store with two books and a bicycle:
{ "store": {
"book": [
{ "author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
},
{ "author": "J. R. R. Tolkien",
"title": "The Lord of the Rings",
"isbn": "0-395-19395-8",
"price": 22.99
}
],
"bicycle": {
"color": "red",
"price": 399
}
}
}
The value is an object containing a single value named “store”. The “store” value is an object containing two values: an array named “book” and an object named “bicycle”. Each book in the array, as well as the bicycle, is an object with various values consisting of strings and numbers. The corresponding tree of nodes is shown below:
JSONPath provides a way of selecting and extracting nodes from a JSON value. For example, the JSONPath query $.store.book[0] applied to the value above selects the first book (by Nigel Rees) whereas the query $.store.book[*].price extracts the prices of books: 8.95 and 22.99 (since [*] selects all the nodes of an array). The query $..price extracts all the prices: 8.95, 22.99, and 399 (although not necessarily in that order; .. selects the nodes contained in a value, known as the descendants of the value)
The result of a query can contain any number of nodes and so is considered to be a, possibly empty, list (known as a nodelist).
The uses of JSONPath include:
Selecting a specific node in a JSON value
Retrieving a set of nodes from a JSON value, based on specific criteria
Navigating through complex JSON values to retrieve the required data.
For more information, please refer to the RFC.
With the basics of JSON and JSONPath covered, the following is my personal perspective as one of the editors of the RFC.
Working Group activity
A few years ago, I couldn't find a JSONPath implementation that documented the syntax and semantics clearly, so I wrote my own. Towards the end of this effort, I decided there was a need for a standard.
In 2020, I settled on the IETF as a standards body and gathered together a small group of JSONPath implementers (found via Christoph Burgmer's excellent JSONPath comparison project). Together we cooked up an initial spec and published it as an IETF Internet-Draft.
Around the same time, Stefan and IETF veteran Carsten Bormann submitted their own draft, based closely on Stefan's original blog post. In September 2020, we joined forces in a Working Group (WG) with the help of Tim, James Gruessing (who co-chaired the WG), and others. After agreeing a charter, we merged our drafts and set about iterating on the spec.
The WG met online and never in person, but I think we got to know each other fairly well by interacting on the mailing list, issues, and pull requests.
The WG needed to support regular expression matching for filtering objects and arrays, so Carsten and Tim defined a subset of most other regular expression flavours (“I-Regexp”) that was ultimately published as RFC 9485.
The WG is now dormant. The mailing list is still open for business. The spec repository lists topics deferred by the WG or raised subsequently. I don't plan to work on another version, although others may.
Sticking to the charter
The WG charter included the following:
The WG will develop a standards-track JSONPath specification that is technically sound and complete, based on the common semantics and other aspects of existing implementations. Where there are differences, the working group will analyze those differences and make choices that rough consensus considers technically best, with an aim toward minimizing disruption among the different JSONPath implementations.
There was a constant tension in trading off technical soundness against common semantics. Where there was a clear consensus among implementations, the spec tended to take that line. But where there were variations in behaviour, we had to choose what we thought made the best technical sense.
At several points, suggestions were made for going beyond the clear consensus. Many of these were captured as issues labelled revisit-after-base-done. In general, we avoided innovation.
There was also pressure to give some implementations, such as the popular Jayway Java implementation, higher precedence than others. But biassing the spec towards particular implementations would not have been consistent with the charter.
Then there was pressure to adopt features unique to particular projects, such as JMESPath which goes way beyond the original blog post, but is carefully thought through and well-documented. Again, this would not have been consistent with the charter.
Extension mechanism
One area where we had to innovate was the extension mechanism for functions inside filter expressions. We wanted it to be possible to define functions without having to change the spec, but there wasn't a common precedent to follow.
The main complexity was defining what constituted a valid, as opposed to a merely syntactically well-formed, function expression. We introduced a type system to define validity, but there was extensive debate around:
subtyping and implicit type conversion,
functions to perform explicit type conversions,
the usefulness of a SingleNodeType type for the result of a singular query (a JSONPath expression that necessarily outputs at most one node), and
whether to attempt to retrofit the type system to the rest of the spec.
To avoid unnecessary complexity, we ditched subtyping and the SingleNodeType, defined just one implicit type conversion (from NodesType to LogicalType), and limited the type system to function expressions.
The final type system, consisting of just three types, seems to work well:
The solid arrows denote “is an instance of”; the dashed arrow denotes the implicit conversion from NodesType to LogicalType.
We also included five functions in the spec, although we expect others to be registered with IANA in due course.
IETF openness
This was my first experience of developing an IETF standard and I was delighted by the truly open approach. As I was accustomed to working on open source projects, the openness felt reassuringly familiar and it was this that initially attracted me. Anyone is welcome to submit an Internet-Draft, get involved in a WG or to provide one-off pieces of feedback. Indeed there is no concept of membership of a WG, or even of the IETF. For example, the JSONPath Interim drafts, mailing lists, and spec repository are all publicly visible.
The journey of a spec through the IETF approval process is transparent: see, for example, the handling of an objection during the last call process for JSONPath and the archive of the Authors’ Final Review (AUTH48) process.
There was also no corporate involvement in the JSONPath WG (although my previous employer, VMware, sponsored some of the early work on the spec, but without interfering in any way). This was refreshing as I had previously represented my employers in other standards bodies and occasionally found the politics tiresome.
Consensus decision-making
Another interesting facet of the IETF was the notion of rough consensus, which helped the Working Group avoid stalemate on a few occasions. My understanding of rough consensus is simply that unanimous agreement isn't always necessary, but where there are objections, the reasons for these objections are explored by the group and a decision taken on the basis of this exploration.
This approach to decision-making suits the IETF and its WGs: since they don't have members, voting would be meaningless. It also avoids “design by committee” where politics impacts technical quality.
Adoption
Now there is a standard JSONPath, it will be interesting to see whether, and how quickly, implementations start to adopt the standard.
The standard already has the following implementations:
As new programming languages are created and need support for JSONPath, hopefully these will be written to conform to the standard.
Another potential use for the standard is in documenting non-standard implementations in terms of which features do, or do not, match the standard. At the very least the standard provides some well-defined terms for discussing JSONPath implementations.
Clearly, with some 50 implementations in the wild, convergence is likely to be a slow process. But the beauty of an IETF standard is that it is a permanent document that should stay the course.
Conclusion
It has been fascinating seeing the development of a RFC from scratch. I am indebted to everyone who worked on this: those who used my Go implementation and reviewed its documentation, those who contributed to a Rust implementation and a test suite (both described below), and, of course, my fellow editors, the rest of the WG, the RFC Editor, and others in the IETF who helped bring the RFC to fruition.
Finally, just for fun, here's an animation showing some of the activity on the spec repository:
(If your browser doesn't support HTML video, here's a link to the video instead.)
Follow on work
There has been good progress in developing a Compliance Test Suite for JSONPath and some possible beginnings of a Reference Implementation, both described below.
Towards a Compliance Test Suite
There is an incomplete Compliance Test Suite (CTS). The most notable omission from my perspective is testing for non-deterministic behaviour. Currently, all the tests in the CTS produce deterministic results. So any valid implementation should pass the CTS. However, the non-deterministic aspects of the spec (which stem from the unordered nature of JSON objects) are not tested by the CTS.
The implementations mentioned above have been tested against the CTS.
Carsten's Ruby implementation of RFC 9485 contains some basic tests, but a full CTS has not been developed. Perhaps a CTS could be based on the XML Schema Test Suite.
Towards a Reference Implementation
I started a Rust implementation that I hoped could be used as a Reference Implementation. However this effort is now dormant since I ended up spending much more time editing the draft than I anticipated and the parsing code, using the pest parser, was surprisingly laborious to implement (compared to my previous hand-crafted parser in Go).
The point of using the pest parser was that the parsing expression grammar (PEG) was cleanly separated from the implementation and was therefore easier to compare to the ABNF in the spec than, for example, a hand-crafted parser.
Carsten's Ruby implementation uses his abnftt tool to generate a PEG parser from the ABNF in the spec. Given that the WG tried to keep the ABNF “PEG-compatible” by ordering choices suitably, the parser is highly likely to match the spec. This may be a more promising approach for a Reference Implementation than manually comparing a hand-coded PEG to the ABNF.
Carsten also wrote a Ruby implementation of RFC 9485, again based on abnftt. The code is available on github. For usage information, issue:
$ gem install iregexp
$ iregexp --help
This is a reference implementation of RFC 9485 only insofar as it can be used to express I-Regexp regular expressions in other popular flavours such as PCRE or JavaScript regular expressions.
Postscript: This post claims the RFC was published “today” (21 February 2024, the date of the post) because of the timezone in which publication occurred and because of the whole number of years since Stefan's blog post. The RFC was actually published in the early hours of the morning of 22 February here in the UK and in Germany (its moral birthplace), but I'm going to resist getting hung up on that.
A variant of this post was also published on the IETF blog.
I'm currently teaching a second year university course on functional programming in Haskell. This will be my second time through. I inherited the slides from a colleague who created the course and taught it once. This time I'm busily converting the slides into Beamer format and taking the opportunity to improve them.
Inevitably, while improving things, I realise I understand some parts, but not others. This struck me particularly when working on the second lecture on Haskell's type system.
The basics
The first mention of type checking in the second lecture uses this example:
ghci> :t 3+4
3+4 :: Num a => a
I explain this as follows:
Num a => a is the type signature of the expression 3+4
Types starting with a lowercase letter, such as a, are type variables
The constraint operator => constrains types in the type signature
So the second line says that the type of 3+4 is any type that implements the Num type class
Haskell doesn't assign a concrete type to 3+4. It can be used as an Int, a Float, etc.
Types vs type signatures
According to Haskell 2010, a type signature has the form:
v₁, ..., vₙ :: cx => t
This declares that the variables v₁, ..., vₙ are of type t with respect to the context cx.
So in my example above, 3+4 :: Num a => a is actually a type signature and specifies that the type of 3+4 is a, for any type a that implements Num. That's also the type of 1, to answer my opening question.
Let's take a short detour to discuss the fact that a constant expression can have more than one type.
Parametric polymorphism and constant expressions
Parametric polymorphism is often understood to apply to functions. For example, the function head that returns the first element of a non-empty list has this type signature¹:
head :: [a] -> a
The function is polymorphic: it can be applied to any list, regardless of the type of the list elements.
But parametric polymorphism covers constants too. According to [1]:
Polymorphism means that one and the same expression can have (perhaps infinitely) many types.
Type inference
The type Num a => a above is inferred by the compiler. The Haskell wiki talks about the compiler inferring concrete types whenever possible, but type inference isn't all about concrete types as our example shows.
Take a slightly more complicated example map incr where:
incr x = x + 1
The compiler infers the type of incr to be Num a => a -> a and the type of map incr to be Num a => [a] -> [a]. What's going on here?
Principal types
Any expression in a valid Haskell 2010 program has a most general type, known as the principal type, that can be assigned to it. Another way of stating this is that Haskell 2010 has the principal type property. In most cases the compiler can infer the principal type of an expression.
Unfortunately, certain GHC extensions, notably Generalised Algebraic Data Types (GADTs) and rank-n types, destroy the principal type property. With such extensions, some expressions do not have a principal type since they would be satisfied by more than one incomparable type. They either require the compiler to choose one of the possible types and/or the programmer to narrow down the choice using type signatures.
Even Haskell 2010 requires type annotations in some cases (most notably for code with otherwise ambiguous types, and for code using polymorphic recursion). Each expression in Haskell 2010 has a principal type, but the compiler cannot discover it in these cases.
[2] Typing Haskell in Haskell by Mark P Jones is a formal, in-depth account of building a Haskell type checker in Haskell.
[3] Type inference for Haskell is a 23-part series of articles by Jeremy Mikkola, which he wrote as a beginner-friendly companion to Jones's paper.
I think there's a gap in market for a description of type inference in Haskell between the level of this post and Mikkola's series, suitable for newcomers to Haskell (like my students). If anyone knows of such a description, I'd be grateful if they would email me via the “contact” link below.
I recently noticed which technical books I can't bear to part with. Here's a photo of the top ten:
The stack is roughly in chronological order of reading from bottom to top.
Mathematical Analysis
“Apostol”, as it's fondly known, deserves a special mention. One of my university tutors recommended reading some of it and working through some exercises in preparation for my degree. I recently met up with him and he confessed the book may have been rather challenging at that stage. However, the first few chapters were gripping and filled me with excitement and anticipation ahead of starting my studies at university.
The Z Notation
A couple of years after starting work as a software developer, I was disappointed by the quality of specifications. I heard about a collaboration between Oxford's Programming Research Group (PRG) and the CICS development group. Eventually, I got to join CICS development and get involved in the collaboration personally. Around that time I went on a course on the Z specification language taught by the PRG and wrote several Z specs.
I continued to use Z from time to time throughout my career. Even when it was just for personal consumption, I found writing a Z spec helped me get a much clearer understanding, either of a piece of software I was developing or of a dependency that wasn't very well documented.
The Theory and Practice of Concurrency
While I was working for IBM, I had the option of doing a modular MSc at Oxford. When I looked at the modules available, I felt two of them, on concurrency would benefit me significantly. The first was on the fundamentals of Tony Hoare's Communicating Sequential Processes (CSP).
The second was Bill Roscoe's course on modelling and checking concurrent systems in CSP using the FDR2 tool. His book describes the Failures/Divergences Refinement model underpinning this tool. The course was a white-knuckle ride of deep theory and challenging practical exercises, but was great fun.
The Elements of Style
This book, given out on a course on technical writing, really helped improve my writing.
tl;dr omit needless words.
Java Concurrency in Practice
I forget the details of much of this book as it's been many years since I wrote any Java. But the description of the Java Memory Model in Chapter 16 was worth the price of the whole book. The mathematician in me particularly liked the description of the happens-before relation as a partial order.
Clean Code
I would say this book greatly improved the clarity of my code. I still have a preference for writing small, well-defined functions, especially as good function names help to make code self-documenting.
Internetworking with TCP/IP (vol. 1)
Until I read this book, my understanding of networking was patchy and poor. Then working on a container manager forced me to get to grips with subnets, IP tables, and the like. After being disappointed by books that tried to explain how to manage networks and write networking code, I was delighted to discover Comer's book that explained the principles of networking in a coherent and complete way.
Programming Rust
I've only written a relatively small amount of Rust code, but whenever I do it's a delightful experience. It's probably not the easiest language to use, but I appreciate the combination of high level features and the suitability of the language for system-level programming of operating systems and the like. To be honest, I don't remember anything from this book, but I think it was one of the best introductions to the language.
Learn You a Haskell for Great Good!
Having read “Real World Haskell” over the Christmas holidays many years ago, I was delighted by the much more accessible approach taken by LYAHFGG and recommend it to my functional programming students.
Programming in Haskell
This is my most recent Haskell book and it surpasses even LYAHFGG in accessibility and clarity. Again, this is a book I recommend my functional programming students. Graham Hutton's Haskell lectures on YouTube are also a delight to behold.
Regular readers will have noticed a recent flurry of activity here. I've been refactoring the pages linked in the footer of each post after thinking about Ken Zinser's post on “cornerstones” of a personal website.
I'm currently supervising some computing students doing their final year project, delivering a module on functional programming (in Haskell), and waiting for the JSONPath RFC (of which I am an editor) to be published.
I'm reading Rory Stewart's excellent political autobiography “Politics On the Edge – A Memoir From Within”.
I had to think long and hard about the chances of degoogling any time soon before including my gmail address in an upcoming RFC.
Although Google products and services are amazing, I'm concerned about the privacy implications and the near monopoly Google has in some areas. So I hesitated because having a gmail address is an implicit endorsement of Google.
I kept it in the RFC since I can't see me degoogling completely any time soon, for the following reasons.
An obvious step towards degoogling involves hiding my email address (be it gmail or something else) behind the custom domain name I purchased just two months ago. But I have little confidence that the vendor will keep the price of the domain name reasonable for the foreseeable future¹.
The best email provider I've found is Protonmail, but the basic paid subscription is a little pricey (remember I'm a Yorkshireman) and the free version doesn't do quite enough.
I have two personal gmail accounts, another I use as a charity Trustee, and another I use for live streaming at my church, although thankfully the latter two are hidden behind custom domain names.
I have (what feels like) countless website, utility, and other accounts linked to my gmail accounts.
I have numerous Google docs that I'd need to migrate, possibly to open office docs on cloud storage. Can I afford the time?
I have an Android Fairphone, which I really like. I wouldn't buy an iPhone for environmental reasons. I don't want the hassle, and risk of incompatibility, of running another OS on my phone, especially as I hope to keep using a phone as I get much older (I'm 63).
I use Google maps for navigation and am not aware of any great equivalent.
My Google calendar inter-operates well between phone, Macbook, and Linux desktop. Not sure I'd find something better.
I love Good authenticator and somewhat loath the Microsoft equivalent.
I switched from Yahoo mail to gmail around 2008 and I still haven't deleted my Yahoo account. I'd probably need to hang on to my gmail accounts for ages too.
So I must congratulate Google on locking me in good and proper. But, given my ad blocking habits (Firefox with uBlock origin), I suspect they don't make much profit from me.
On a positive note, I switched from Google Chrome to Firefox a while back and from Google Search to DuckDuckGo in the last year or so and haven't missed either of them.
¹ Thanks to feedback from Benjamin Hollon, I clarified that it's the price of the domain name, and not the email provider, that concerns me on this point.
| about | blogroll | contact | now | popular | subscribe | 
]
