How to use reader macros to enhance Lisp's expressive power

One of my favourite programming languages is Common Lisp. Lisp's most distinguishing capability is its powerful macro system, and a distinctive feature of this system is the reader macro facility. Reader macros give you control over Lisp's parser so that you can manipulate how it interprets source code. Here's how I've written various reader macros to add syntax and adapt existing syntax, allowing me to write more expressive code.

OfferZen_How-to-Use-Reader-Macros-to-Enhance-Lisps-Expressive-Power_inner-article

Reader macros in Lisp

Most programming languages don't let you configure how the parser interprets raw text, but Lisp does, using reader macros.

Lisp has two mechanisms for adding a reader macro:

set-macro-character
set-dispatch-macro-character.

Using set-macro-character you can specify a single character as the beginning of a reader macro. The problem is that there's a limited supply of special characters to use for this. For this reason, most reader macros use set-dispatch-macro-character, which uses two characters. With two characters instead of one, many reader macros can be defined using just one special character. By convention, reader macros defined with set-macro-dispatch-character use the # character as the first character with an arbitrary second character, for example, #r or #c.

Both of these mechanisms add a hook to the Lisp parser, and when it encounters the specified character or character sequence, it hands over parsing to a function you provide. Your function reads the input stream until it returns, and then the Lisp parser takes over again.

How I've used reader macros in Lisp

I've used reader macros to provide a shorthand for lambda expressions that reduces the amount of code I need to write. They've also been useful for incorporating ideas from other languages like currying from Haskell, which improves the composability of functions, and decorators from Python that can be used to modify the behaviour of functions. Finally, I've found reader macros useful for writing values in a natural way. A fun use case for this was adding literal Roman numerals to Lisp, which I'll cover below.

Abbreviating lambda expressions

Common Lisp's syntax for lambda expressions can be heavy when you only need to invoke a simple function. For example, if you want to add one to every element in a list, you'd write something like this.

  (mapcar (lambda (n) (+ n 1)) numbers)

This maps a function that adds 1 to each element in the list numbers and returns a new list. However, having to write out (lambda (n) ...) to create this anonymous function seems like a lot of code to express something simple, so I wrote a reader macro.

When using reader macros you don't necessarily need to write a parser from scratch. You can use Lisp's built-in read function to parse the next Lisp form or symbol in the input stream. I wanted a simple way of writing lambda expressions where I didn't have to write out the full lambda definition with its arguments. This is what I was after:

  (mapcar #l(+ %n 1) numbers)

In this case, I decided to use set-macro-dispatch-character using #l as the character sequence that tells the Lisp parser to hand over control. I defined a function to take over the parsing responsibilities called #l-reader.

  ;; Parsing function
 (defun |#l-reader| 
    (stream subchar arg)
    (declare (ignore subchar arg))
    (let ((exp (read stream))) 
      (expand-lambda exp))) 
  
  ;; Reader macro hook 
  (set-dispatch-macro-character #\# #\l #'|#l-reader|)

I used the built-in Lisp read function to read the next Lisp form, which is the body of the lambda expression. In my example, this is:

  (+ %n 1)

Then I use my expand-lambda function to convert that Lisp form into the lambda body that I would have written by hand. The result looks like this:

  ;; result of applying expand-lambda to (+ %n 1)

  (lambda (%n) (+ %n 1))

This Lisp form is what is returned by my parsing function to the Lisp compiler, which turns it into an actual anonymous function that can be used by my code.

With this reader macro, I can write anonymous functions concisely. This saves keystrokes and makes my code clearer by eliminating unnecessary details.

Currying functions to improve composability

Often it's convenient to convert a function expecting many arguments into a function that only needs a few, or only one. In functional programming languages from the ML family, such as Haskell, OCaml and FSharp, this is called currying, and there is syntax to support it.

Currying works by supplying some of the arguments to a function and returning a function that only needs the remaining arguments. It's very useful for composing functions. Currying is also useful for common functional programming operations like map and filter because they also expect one argument functions. For example, if I wanted to remove all occurrences of the number 5 from a list in Lisp, I'd use the remove-if function along with a lambda.

  (remove-if (lambda (n) (= n 5)) numbers)

In Haskell, if you're using currying, there's no need to explicitly create an anonymous function. Instead, you can use currying to partially apply the /= not equal function.

  filter (/= 5) numbers

I wanted to be able to do this in Lisp too. So I added a reader macro for partial function application, #p:

  (remove-if #p(= 5) numbers)

The #p reader reads the next Lisp form and constructs a lambda expression which remembers the arguments supplied so far and applies the function to the remaining arguments. In this example, the generated code for the lambda looks like this:

  (lambda (&rest arguments) (apply #'= 5 arguments))

Here apply calls the = function with the argument 5 as well as the additional arguments supplied to the lambda expression.

Python style decorators

A common experience among Lisp users is to notice a feature from another language that looks nice and add it to Lisp. Version 3 of Python, for example, added decorators which allow you to modify a function's behaviour. Memoization can be added to a Python function using the @cache decorator. The Fibonacci function in Python using a decorator for caching looks like this:

  @cache
  def fib(n): 
    if n == 0:
      return 0
    elif n == 1:
      return 1 
    else: 
     return fib(n-1) + fib(n-2)

Doing something similar in Lisp would involve rebinding the function name to a version of the function which caches values in a hash table. This can be encapsulated in a memoize function. I defined a reader macro using @ as its hook character. It reads a function name and a lisp function body and applies the named function to the function body. I implemented this reader macro myself, but a similar reader macro for decorators is available in the cl-annot library.

  (defun |@-reader| 
       (stream char)
       (declare (ignore char))
       (let ((decorator (read stream))
              (fdef (read stream)))
          `(progn ,fdef (funcall #',decorator ',(cadr fdef)))))

By using this reader macro and the memoize function, the same caching behaviour can be added to Lisp functions.

  @memoize 
  (defun fib (n) 
    (cond ((= n 0) 0)
             ((= n 1) 1)
             (t (+ (fib (- n 1)) (fib (- n 2))))))

Adding Literals for Roman Numerals

Another example of something you can do with reader macros is adding literals. For example, you could add literal inline JSON notation or literal representations for types from your domain. As a fun exercise, I used reader macros to add literals for Roman numerals. The reader macro dispatch sequence I used was #r.

My parsing function parse-numeral converts a sequence of Roman numeral characters into an integer so that Roman numeral literals can be used like ordinary Lisp numbers.

  (defun |#r-reader|
      (stream subchar arg)
      (declare (ignore subchar arg))
      (let ((numeral (map 'string #'identity 
                 (loop for c = (peek-char nil stream nil) 
                          while (not (whitespacep (numeral-value c))) 
                          collect (read-char stream)))))
           (parse-numeral numeral)))

With this reader macro, I can use Roman numerals in my code as ordinary values.

  (reduce #'+ (list #rI #rIX #rLXX)) 

  ;; CL-USER> 80

A unique dimension of extensibility

By giving you access to the parser, Common Lisp's reader macros allow you to invent arbitrary syntax. This adds a dimension of extensibility that is unique to Lisp. A lot of the time, it makes sense to leverage Lisp's built-in parsing functions, such as read, and not to radically depart from Lisp's syntax. It is easy to adapt Lisp's syntax enabling neat simplifications and shorthand that reduces boilerplate code.

Reader macros that I've written give me a toolkit that saves me time by allowing me to write less code. Using these reader macros, I can also express myself more naturally. They make my code more composable and less cluttered with irrelevant details.

Henry is a Scala engineer at GrapheneDB where he uses functional programming to manage graph databases in the cloud. He loves functional programming because it makes code robust and easy to reason about.