Frogs are goated

After my switch from vim to emacs and my following rediscovery of common lisp, I started to write a few tools in emacs lisp so I could use them directly from inside my editor. Eventually I started implementing a few of them in common lisp instead, so I could use the better programming facilities that common lisp provided me with compared to elisp.

Using the excellent development environment known as Superior Lisp Interaction Mode for Emace (SLIME) in combination with Steel Bank Common Lisp (SBCL) I happily hacked away at a few challenging tasks.

One day, I came to the conclusion that I needed header information from a certain binary file format my company uses so often, that it was sensible to write a program that provided the ability to "grep" header info from such a file. The program that originated from that problem was called grep-bin and could be invoked like so: (grep-bin "filename.bin" 'version 'date) to extract the version entry as well as the creation date from the binary file.

Often times my company also uses remote procedure call servers running on certain hardware, to bridge the hardware registers to the network. I won't go into more detail than that here, but I got the idea that for development it could be very useful if such a server was running in lisp, because it would enable me to send lisp snippets to such a server to evaluate. Once development was finished, it could then be compiled to machine code and stay on the hardware indefinitely. Also the remote code execution should be removed for production…

So I set out to write lisp-rpc in Common Lisp which as it turns out was not difficult at all. Using the popular libraries usocket and bordeaux-thread, it was just a matter of around 250 lines of lisp to implement a minimum feature set, including sending lisp code, writing all supported remote procedures into a lisp file on shutdown and loading it on startup if it existed.

The real challenge would begin now: Getting the programs to run on real hardware. At first, I tried to simply compile the file as a binary using SBCL, which did work, and resulted in an 11 MB ELF file, which was only ever going to run on my x86₆₄ linux machine. The target hardware however was an embedded SoC made by in-circuit. As that arm32 board was relatively limited resource-wise, I opted to not throw an 11 MB binary on it, even if that would ultimately result in a solved problem. Instead I browsed the web to find another Common Lisp implementation, which would aid me better in my specific endeavours.

To the rescue came Embeddable Common Lisp (ECL), which advertises to run on any platform providing at least a 32 bit C compiler. As I already had a buildroot based toolchain which worked perfectly fine for the hardware board, I set out to cross compile ECL and then my programs.

After fiddling a little bit with the configure script of ECL's autotools based build system with which I was not very familiar, I managed to build ECL natively so it would replace SBCL in SLIME for the time being. Next, I managed to point it to my buildroot toolchain and after a couple of failed attempts, it would create a working cross ECL.

Although ECL worked on the embedded board, it again was one big ELF file, statically linked to libglm and libffi. I managed to wrangle the configure script enough to create a dynamically linked version of it and proceeded to create a compiled version of the grep-bin program.

Now both, ECL as well as grep-bin linked dynamically to the afforementioned libraries. grep-bin worked, but had six seconds of startup time. But it worked. This was a major milestone for me, as I managed to cross compile a Common Lisp program to arm32.

Due to the startup time and due to grep-bin being a simple one-off program that no one wants to spend six seconds waiting for, I ended up rewriting it in C and calling it a day. Now its fast and small and unsafe and does not link to either libgmp nor libffi. But its not Common Lisp, either.

Even though I ended up rewriting grep-bin in C, I am grateful to have tried doing it in Common Lisp and using ECL. The learning was fun and the IRC channel #ecl was always a pleasure to consult with.

The biggest pitfalls for me in this step have been:

• learning how to configure properly
• learning to put the cross toolchain into PATH when calling configure and make, source of many a confusion
• copying the cross target-info.lsp into a directory where it would stay
• learning that all the C API of ECL was not important for my use case at all, except as a mental model how the files would be translated along the way.

With the easier part out of the way, I tried to get lisp-rpc to run. As it turned out this was much more cumbersome due to its dependencies. First, I tried the most simple looking approach: just get asdf cross compiled to use it so it was as simple as (asdf:load-system :lisp-rpc) . Unfortunately I ran into a few problems doing so, because for example usocket which I used as a TCP socket library needed to be natively compiled for a few functions that I used.

Trying to compile usocket was not as easy as I hoped for. First, when I attempted this, ECL did not yet have a way to cross compile entire systems. Luckily for me, there was this project, that jackdaniel in #ecl could provide me with, that aimed to enable cross compiling to web assembly. After a few tweaks it was functional to compile systems using my cross toolchain.

I moved on to only natively compile usocket as a shared object but it didn't even occur to me that I would have no way to load an .so file from my running ECL session, so naturally that was the next issue I ran into head first. After jackdaniel nudged me into the right direction, I managed to compile usocket as a .fasl file which, according to the ECL documentation is a shared object but it has a bit of code around it that makes it loadable from lisp. In other words, exactly what I was looking for.

The next step was to also cross compile bordeaux-threads the same way I did with usocket but I got compilation errors along the way that didn't really tell me much. The error message was something along the lines of "unexpected EOF in SYS:help.doc". I opened the source file in bordeaux-threads that was causing the compilation error. It turned out to be the very first macro defdfun in the file bordeaux-threads-v0.9.4/apiv1/default-implementations.lisp. This macro tried to use defun in an (arguably) unorthodox way:

(defmacro defdfun (name args doc &body body)
  `(eval-when (:compile-toplevel :load-toplevel :execute)
     (unless (fboundp ',name)
       (defun ,name ,args ,@body))
     (setf (documentation ',name 'function)
           (or (documentation ',name 'function) ,doc))))

Using this macro to create function definitions, a docstring has to be provided (it may be empty though). But here things get a little bit confusing, because this macro is technically doing nothing wrong. It works totally fine when using sbcl for example. After conducting a little bit of research, I found out that the issue was the setting of the docstring the way it was used in this macro. Apparently, when doing this in ECL, the docstring is not saved persistently. This could be demonstrated using just a few lines of code in the repl:

[erru@lispy ecl-26.5.5]$ ecl --norc
ECL (Embeddable Common-Lisp) 26.5.5 (git:UNKNOWN)
Copyright (C) 1984 Taiichi Yuasa and Masami Hagiya
Copyright (C) 1993 Giuseppe Attardi
Copyright (C) 2013 Juan J. Garcia-Ripoll
Copyright (C) 2018 Daniel Kochmanski
Copyright (C) 2024 Daniel Kochmanski and Marius Gerbershagen
ECL is free software, and you are welcome to redistribute it
under certain conditions; see file 'Copyright' for details.
Type :h for Help.
Top level in: #<process TOP-LEVEL 0xb6878f80>.
> (defun foo () "Foo bar." 'foo-bar)

FOO
> (documentation 'foo 'function)

"Foo bar."
> (defun bar () 'bar-foo)

BAR
> (documentation 'bar 'function)

NIL
> (setf (documentation 'bar 'function) "bar.")

"bar."
> (documentation 'bar 'function)

NIL
> 

I let jackdaniel know of this issue and patched bordeaux-threads such that it compiled with ECL:

(defmacro defdfun (name args doc &body body)
  `(eval-when (:compile-toplevel :load-toplevel :execute)
     (unless (fboundp ',name)
       (defun ,name ,args ,doc ,@body))))

Now I could cross compile bordeaux-threads without issues, since the system compilation script took care of system dependencies.

The final step was actually getting lisp-rpc to run. After loading all the libraries which took a few minutes on my embedded system, it actually started up fine and even ran reasonably fast.

I didn't do a good job benchmarking and comparing to my C++ server, however. What I did notice though, was that the lisp server consumed about 5.5 times as much memory as the C++ server (both at idle). That could probably be due to the much more complete environment that lisp runs in. It has a working interpreter, REPL, GC and such right there, all things that are missing in the C++ variant. It uses libraries other than the libc which also does not happen in the C++ variant. The disk usage mirrored that as well, with the C++ server taking up 100kB as a single executable (excluding the about 5MB used by glibc) while lisp-rpc took up 300kB when compiled (excluding the about 11MB that ECL needed).

The main benefit of lisp over C++ in this case is probably having macros and a fuller ecosystem as well as being an interpretable language that is capable of parsing itself efficiently. But it has to be weighed to the upsides that C++ brings to the table in the embedded scenario as well. While typically troublesome in lisp, C++ provides very easy direct access to hardware (which can be important in embedded systems). It has little to no overhead, executables have fast startup times and it is easier and faster to get source code compiled to the hardware.

Even though it is possible to reach faster startup times and smaller source files for lisp as well via investing a little more time into it, C/C++ already has established itself within the POSIX environment and gained quite a foothold as a consequence. The glibc variant already offers most things that are needed for application development in the embedded domain.

While I enjoyed trying out lisp's cross compilation capabilities, I have to conclude that a rewrite of a perfectly fine C/C++ program to a lisp one is probably not worth it.

That being said, for initial development, it is feasible to have lisp-rpc running on the hardware that can interpret and run code snippets that a developer sends to it. This makes initial embedded development significantly easier. Embedded software written in this way has a high chance of staying the way it is because rewriting it is an expensive endeavor, so if a new project starts out like this, I think that it could just stay written in lisp the way it is.

I would love to see lisp as a more prevalent language in use, but I think that the embedded systems domain is probably one of the hardest and most thankless ones to really get into, not just for developing lisp but also for developing applications in general. This is further amplified by embedded software engineers learning primarily C/C++ and not lisp (I learned lisp just because its fascinating).

On the flipside, when developing software for the x86_64 platform, I think that lisp can be a fantastic choice, because all the afforementioned pain points kind of fall flat: The issue of the 15MB essentially completely disappears. The startup time and runtime of lisp in comparison to C/C++ can be very similar, because CPUs are just that much faster and have more cores than in the embedded systems domain. But lisp can also reach comparable performance to C/C++, especially when telling the lisp compiler to go fast by providing type information where significant. Lots of effort was put into getting lisp fast back in the hayday of the AI bubble in the 1980s and even today, some lisps push the boundaries by have SIMD instructions covered while some C compiler vendors don't yet.

Overall I very much enjoyed the time I was able to dedicate to this project and I hope to be able to publish the code of the two programs I wrote at some point.

Thanks to all who helped me on this journey, especially thanks to jackdaniel and beach on irc.libera.chat in #ecl and #clschool respectively. You two have been not only extremely helpful but also very nice and encouraging all the time.