Adding JSON literal syntax to Common Lisp

May 2, 2025

Here's a small example of Common Lisp's (in?)famous extensibility.

Table of Contents

TL;DR

By the end of this post, we'll be able to write JSON objects literally in Lisp code with the ability to mix syntax: 

CL-USER> (defvar a-key "x")
CL-USER> (defvar a-value 3)
CL-USER> {
    :w : nil,
    a-key : 40,
    "y": true,
    "z": {
        "a": [1, 2, a-value],
        "b": #("a" "b" "c")
    }
}

The above code will evaluate to a pretty-printed1 JSON object: 

#<JSON-OBJECT {
  "w": false,
  "x": 40,
  "y": true,
  "z": {
    "a": [1, 2, 3 ],
    "b": ["a", "b", "c"]
  }
} {1017F09343}>

We'll also have a saner lisp-y way to do the same: 

(json-object
 :w nil
 a-key 40
 "y" t
 "z" (json-object "a" (json-array 1 2 a-value)
                  "b" (json-array "a" "b" "c")))

If you want to skip the post and just get the code, you can find it here.

JSON library

I really like com.inuoe.jzon (jzon). Its philosophy is to adhere to standards and correct behavior as much as possible, and it's fairly straightforward to use.

It uses hash-tables as the main representation of JSON objects, which is quite a reasonable and practical design choice, but it also means that:

  1. We don't get a dedicated JSON object.
  2. It's bit cumbersome to create objects on the fly.

The first is not always desirable, and the second is actually just a limitation of standard Common Lisp (the absence of hash-table literals), and can be solved a number of ways, my favorite of which is just to use alexandria:alist-hash-table: 

(alexandria:alist-hash-table '(("key1" . "value1") ("key2" . "value2")) :test #'equal)

Still, sometimes we may want to distinguish between a "regular" hash table and one meant as a JSON object. Let's write a small utility for that.

Helper class and functions

Let's define dedicated classes for JSON objects (and while we're at it, JSON arrays). These will just wrap the objects jzon uses for its representation, which are hash-tables and vectors respectively: 

(defclass json ()
  ((object :initarg :object
           :accessor json-lisp-object))
  (:documentation "Parent JSON class"))

(defclass json-object (json) ()
  (:documentation "JSON object wrapper for hash-tables."))

(defclass json-array (json) ()
  (:documentation "JSON array wrapper for lisp vectors"))

Then we'll define a specialization of com.inuoe.jzon:write-value on our new class, so that jzon knows how to serialize it: 

(defmethod com.inuoe.jzon:write-value ((writer com.inuoe.jzon:writer) (json json))
  "Serialize underlying JSON lisp object to json."
  (com.inuoe.jzon:write-value writer (json-lisp-object json)))

We'll also need an object printer, so we get immediate feedback on how our JSON object will look at the REPL: 

(defmethod print-object ((json json) stream)
  "Print Lisp JSON object, showing what it would look when serialized to JSON."
  (let ((string (com.inuoe.jzon:stringify json :pretty *print-pretty*)))
    (if *print-readably*
        (error "No readable printer implemented for json object")
        (print-unreadable-object (json stream :type t :identity t)
          (princ string stream)))))

Finally, some utility fuctions to easily create JSON objects without having to do the song and dance of creating hash tables manually: 

(defun json-object (&rest rest)
  "Create new JSON object using REST arguments as key value pairs."
  (let ((ht (make-hash-table :test #'equal)))
    (when rest
      (loop :for (key value) :on rest
              :by (lambda (list)
                    (if (null (cdr list))
                        (error "Missing value for key: ~a" (car list))
                        (cddr list)))
            :do (setf (gethash key ht) value)))
    (make-instance 'json-object :object ht)))

(defun json-array (&rest rest)
  "Create new JSON array using REST arguments as array elements"
  (make-instance 'json-array :object (coerce rest 'vector)))

Here's what the usage looks like: 

CL-USER> (json-object :x 40 :y "20")

#<JSON-OBJECT {
  "x": 40,
  "y": "20"
} {100E88CB73}>

CL-USER> (json-array 1 2 3)

#<JSON-ARRAY [1, 2, 3] {100EB34AA3}>

CL-USER> (json-object
          :x 40
          "y" 20
          :z (json-object
              "a"
              (json-array 4 5 6)))

#<JSON-OBJECT {
  "x": 40,
  "y": 20,
  "z": {
    "a": [ 4, 5, 6]
  }
} {100EBCDD33}>

That's neat, and it would make sense to stop here. That said…

Relating JSON and Lisp syntax

JSON's standard syntax is pretty minimal:

numbers
integers or floats
strings
double-quoted strings
objects
delimited by {}, use : to separate keys and values, and use , to separate pairs.
array
delimited by [] and use , to separate values.
boolean
true or false
null
null

Numbers and strings are essentially already represented in exactly the same way in Lisp. Booleans and null can be represented as Lisp symbols.

Objects and arrays are where things gets interesting. {} and [] are just regular symbol identifiers in Lisp with no special syntactic meaning. In fact, they're "explicitly reserved to the programmer." They're also very seldom used, so… why not define special syntax for these symbols so we can write JSON directly in Lisp code?

I'm not saying it's a great idea, I'm just saying it's possible with standard Common Lisp.

Implementation

To do this, we'll use macro characters. The idea behind a macro character is that when the Lisp reader encounters it, a function is called that has access to the full stream of remaining code and can read and replace whatever it wants, however it wants.

We can use set-macro-character to define [ and { as macro characters by passing it the character and then a function to handle the syntax. The function is passed two arguments: the stream after the character, and the character itself.

First, a helper function. This will turn the symbols true, false, and null (at read time) into the values that jzon expects, and leave everything else as-is: 

(defun maybe-json-constant (value)
  "Convert read-time symbols TRUE, FALSE, and NULL to T, NIL, and quoted 'NULL"
  (typecase value
    (symbol
     (cond ((string= (symbol-name value) "TRUE")
            t)
           ((string= (symbol-name value) "FALSE")
            nil)
           ((string= (symbol-name value) "NULL")
            '(quote null))
           (t value)))
    (t value)))

Parsing array syntax

Then we define the function to attach to the [ macro character that translates JSON array syntax into lisp: 

(defun json-array-syntax (stream char)
  "Parse JSON array syntax using macro character."
  (declare (ignore char))
  (let ((*readtable* (copy-readtable))
        (eof (gensym))
        (square-close ']))
    (set-macro-character #\] (lambda (s c) (declare (ignore s c)) square-close))
    `(json-array
      ,@(loop
          :as value := (maybe-json-constant (read stream nil eof))
          :do ;; empty array, []
              (cond ((eq value eof)
                     (error "Expected close bracket or key"))
                    ((eql value square-close)
                     (return args)))
          :collect value :into args
          :do ;; ensure values are followed by a comma or close
              (peek-char t stream nil eof)
              (let ((next (read-char stream nil eof)))
                (assert (or (eql next #\,)
                            (eql next #\]))
                        ()
                        "Expected a comma or an end bracket after value: ~s" value)
                (cond ((eql next #\,)
                       ;; detect trailing comma
                       (assert (not (eql #\] (peek-char t stream nil eof)))
                               ()
                               "Expected a value after comma: ~s , " value))
                      ;; detect close object
                      ((eql next #\])
                       (return args))))))))

First it locally sets ] to a terminating macro character since it's a special character needed for JSON array syntax. Otherwise, ] would be read as part of a symbol instead of as its own entity.

Then it constructs a list of elements by:

  1. Reading a value
  2. Return an empty list if the value is a close bracket, or signalling an error if it's not closed
  3. Collecting the value into a list
  4. Signal an error if a comma or a close bracket don't immediately follow the value
  5. Signal an error if there's a trailing comma immediately followed by a close bracket
  6. Return the value list if the array is closed
  7. Start over at 1

The list is then passed as the arguments to our previous json-array function.

Parsing object syntax

We can do something similar (though slightly more complicated) for JSON objects. This function will be the macro character function of {: 

(defun json-object-syntax (stream char)
  "Parse JSON object syntax using macro character."
  (declare (ignore char))
  (let ((*readtable* (copy-readtable))
        (eof (gensym))
        (curly-close '}))
    (set-macro-character #\} (lambda (s c) (declare (ignore s c)) curly-close))
    `(json-object
      ,@(loop
          ;; read the key
          :as key := (read stream nil eof)
          :as value := (progn
                         (cond ((eq key eof) ; unclosed object
                                (error "Expected close bracket or key"))
                               ;; empty object, {}
                               ((eq key '})
                                ;; TODO: if there was a comma...
                                (return args)))
                         ;; ensure there's a colon after the key
                         (peek-char t stream nil eof)
                         (assert (eql (read-char stream nil eof) #\:)
                                 ()
                                 "Expected a colon after key: ~s" key)
                         ;; ensure we have an actual value and not an eof, close, or comma
                         (let ((next-char (peek-char t stream nil eof)))
                           (assert (not (or (eql next-char #\})
                                            (eql next-char #\,)
                                            (eq next-char eof)))
                                   ()
                                   "Expected a value after colon: ~s :" key))
                         ;; read value
                         (maybe-json-constant (read stream nil eof)))
          :collect key :into args
          :collect value :into args
          :do ;; ensure values are followed by a comma or close
              (peek-char t stream nil eof)
              (let ((next (read-char stream nil eof)))
                (assert (or (eql next #\,)
                            (eql next #\}))
                        ()
                        "Expected a comma or an end bracket after pair: ~s : ~s" key value)
                (cond ((eql next #\,)
                       ;; detect trailing comma
                       (assert (not (eql #\} (peek-char t stream nil eof)))
                               ()
                               "Expected a key after comma: ~s : ~s, " key value))
                      ;; detect close object
                      ((eql next #\})
                       (return args))))))))

It's a similar loop as the array, except it has to read a key, then a colon, then a value, and check for unexpected characters or unclosed brackets as well. The list of keys and values is passed to our earlier json-object function.

On/off switch

The last thing we need is a way to turn this on and off. It's generally good practice not to define funky, custom macro characters globally, so we can sandwich code that uses them in the following functions: 

(let ((square-macro)
      (curly-macro))
  (defun enable-json-syntax ()
    (setf square-macro (get-macro-character #\[)
          curly-macro (get-macro-character #\{))
    (set-macro-character #\[ #'json-array-syntax)
    (set-macro-character #\{ #'json-object-syntax))

  (defun disable-json-syntax ()
    (set-macro-character #\[ square-macro)
    (set-macro-character #\{ curly-macro)))
  • enable-json-syntax stores any existing macro functions that [ and { may have had attached to them, and sets them as macro characters with our new functions.
  • disable-json-syntax restores the previous macro functions to those characters, which in most cases wil have been NIL, meaning these characters go back to being symbol constituents as in normal Lisp.

Result

Perfect. Now we can literally copy paste JSON into Lisp files or the REPL, and it will be converted into a JSON object! 

CL-USER> (enable-json-syntax)
CL-USER> [
  {
    "x": ["giraffes", "orangutans", "monkeys"],
    "y": [20, 14, 23],
    "type": "bar"
  }
]

#<JSON-ARRAY [
  {
    "x": [ "giraffes", "orangutans", "monkeys"],
    "y": [ 20, 14, 23],
    "type": "bar"
  }
] {100D184A43}>

We can also substitute most JSON components for Lisp literals or expressions. The following code will produce the same result as above2: 

CL-USER> (defvar animals (json-array "giraffes" "orangutans" "monkeys"))
CL-USER> [
  {
    "x": animals,
    "y": #(20, 14, 23)
    :type : "bar"
  }
]

I just think that's fun.

Footnotes:

1

JZON by default prints array elements on their own line. I've manually collapsed them in the results here for brevity.

2

Note that a colon is still a symbol constituent in this example. To use symbols as keys and values, you'll need to separate the symbol from the colon with a space. Otherwise the Lisp reader would read it all in one go and likely signal errors about suspcious usage of colons in symbols, or nonexistent packages.