Tag Archives: Lisp

Building a Common Lisp program (ECL)

LISP is easy – you just need to start up the interpreter and start playing. But what if you are dependent on libraries, and you want to compile a binary? If you come from another background, like I did, it is quite confusing in the beginning.

Everything below applies to ECL – I think most things will apply to other Common Lisp implementations as well. I build a little command line utility that converts things to/from Base64 (using a library for that).

ASDF
ASDF is a tool that handles dependencies between packages, and also controls your build process (like make). Every project is called a System. Yours to.

When you download lisp packages they typically come with an asd-file, and one or more lisp-files. Each package goes in its own directory, and ASDF needs to know about each package.

I did everything from scratch and installed ecl in /opt/ecl. I put the packages in /opt/ecl/packages (not standard at all).

Project files and building it
These are the files my project contains, and how to build.

kvaser@kvaser:~/lisp/simple-base64$ ls -l
-rwxr-xr-x 1 kvaser kvaser  337 Mar 13 10:18 build.lisp
-rw-r--r-- 1 kvaser kvaser  197 Mar 13 10:18 simple-base64.asd
-rwxr-xr-x 1 kvaser kvaser 1389 Mar 13 13:51 simple-base64.lisp
kvaser@kvaser:~/lisp/simple-base64$ ./build
  ... ...

The binaries (of my program, and all dependencies) end up ~/.cache/, so thats where you need to go to execute your program (or just make a symbolic link to the project directory).

simple-base64.asd

(in-package :asdf)

(defsystem :simple-base64
  :name "simple-base64"
  :author "Zo0ok"
  :version "0"

  :components((:file "simple-base64"))

  :depends-on (:s-base64 :flexi-streams))

:components points to my lisp-file(s).
:depends-on lists other systems that I depend on (the base64-library itself, and a stream library that turned out to be useful.

simple-base64.lisp
Here is the source code to the program itself. It is very non-Lispy, remember, I am new to Lisp and I dont know how to program Lisp with style.


(defun print-usage-and-quit ()
  (format *error-output* "Usage:~%")
  (format *error-output* "  ./simple-base64 -e PLAINDATA~%")
  (format *error-output* "  ./simple-base64 -d BASE64DATA~%")
  (format *error-output* "  ./simple-base64 -e < plaindata.file~%")
  (format *error-output* "  ./simple-base64 -d < base64data.file~%")
  (quit)
)

;;; MAIN starts here

(let ( (mode-op-enc NIL)
       (mode-src-stdin NIL)
       (input-stream NIL)
       (output-stream NIL) )
  (cond
    ( (= 2 (length si::*command-args*) )
      (setf mode-src-stdin T ) )
    ( (= 3 (length si::*command-args*) )
      (setf mode-src-stdin NIL ) )
    ( T
      (print-usage-and-quit) ) )
  (cond
    ( (string= "-d" (second si::*command-args*) )
      (setf mode-op-enc NIL) )
    ( (string= "-e" (second si::*command-args*) )
      (setf mode-op-enc T) )
    ( T
      (print-usage-and-quit) ) )

  (cond
    ( mode-src-stdin
      ( setf input-stream *standard-input* ))
    ( mode-op-enc
      ( setf input-stream (flexi-streams:make-in-memory-input-stream
                         (map 'vector #'char-code (third si::*command-args*)))))
    ( ( not mode-op-enc )
      ( setf input-stream (make-string-input-stream (third si::*command-args*))))
  )

  (if mode-op-enc
    (s-base64:encode-base64 input-stream *standard-output*)
    (s-base64:decode-base64 input-stream *standard-output*) )
)
   
(quit)

Notice that nowhere the systems I depend on are included, they are just used when needed.

build.lisp
Finally the build-script, a lisp program that uses asdf:

#!/opt/ecl/bin/ecl -shell

(require 'asdf)
(push (truename #P"/opt/ecl/packages/s-base64") asdf:*central-registry*)
(push (truename #P"/opt/ecl/packages/cl-trivial-gray-streams") asdf:*central-registry*)
(push (truename #P"/opt/ecl/packages/flexi-streams-1.0.7") asdf:*central-registry*)
(asdf:make-build :simple-base64 :type :program)

Note that the build-script is the place to put paths to systems I depend on. Also note that I have included cl-trivial-gray-streams, a system I dont use directly, but flexi-streams needs it so I need to tell where it is. Finally, this pushing paths to *central-registry* is supposed to be the "old way". But for now I was happy to find a way that works, and that I understand.

Conclusion
As usual, when something works it looks simple, but it is tricky to get all the details right in the first place. I believe this is a good starting point for a small lisp-project that depends on available libraries.

Trivial Gray Streams
The package Trivial Gray Streams caused problems. The standard package I downloaded did not work for ECL (complained it could not find the system). I ended up installing Trivial Gray Streams using Debian apt-get. It puts lisp packages in /usr/share/common-lisp, and that version worked.

This applies to ECL version 11.1.1 and Trivial Gray Streams from Debian 6.0. The version of Trivial Gray Streams that did not work was dated 2008-11-02.

Lisp on Debian/ARM

After reading Revenge of the Nerds I decided it was time to learn Lisp. Programming without some kind of real project is boring, so my plan is to write some web applications using jquery and Lisp (for the back end).

Since I have a Qnap TS-109 running 24×7 I thought it would make a good development machine and Lisp web server. It runs Debian 6.0, but running Lisp on it turned out to be a challenge.

Debian, Lisp and ASDF
Debian supports installing different implementations of (Common) Lisp. However, it seems to be tricky to find a version that installs a binary on Debian ARM.

Also, there is a package depency tool for lisp called ASDF. Lisp implementations should come with it.

GCL
The only Common Lisp that I managed to easily install (i.e. with apt-get) in Debain 6.0 ARM was GCL. But it is a version of GCL that is 5 years old, and it does not come with ASDF.

clisp
I spent much time trying to compile clisp, but in the end I ended up with:

  > ( / 6 3)
  2
  > ( / 5 2)
  Segmentation Fault

Not so fun. Significant parts of clisp is written in assembly (both a good thing and a bad thing), and I was really not able to figure out if it was supposed to work on ARM EABI at all, or just on the old ARM ABI. So after much struggle I gave up clisp.

ECL
I managed to compile ECL from source. Not completely without hassle though. It comes with libffi, but I ended up with compilation errors (the processor does not support…). So, I downloaded libffi, compiled it myself and installed it in /opt/libffi. That was no problem, but I ended up making a symbolic link to include myself:

kvaser@kvaser:/opt/libffi$ ls -l
total 8
lrwxrwxrwx 1 root root   40 Mar 11 16:52 include -> /opt/libffi/lib/libffi-3.0.10rc9/include
drwxr-xr-x 4 root root 4096 Mar 11 16:37 lib
drwxr-xr-x 4 root root 4096 Mar 11 16:37 share

Now I configured ecl with:

CPPFLAGS=-I/opt/libffi/include LDFLAGS=-L/opt/libffi/lib ./configure --prefix=/opt/ecl --with-dffi=auto

That worked, and compiling went fine until ecl_min could not be executed, because it could not find libffi.so.6. I tried to fix that a while, but finally ended up making another symbolic link:

kvaser@kvaser:/usr/lib$ ls -l libffi.so.6
lrwxrwxrwx 1 root root 31 Mar 11 19:56 libffi.so.6 -> /opt/libffi/lib/libffi.so.6.0.0

After that, I ran make again to finish compilation. It went fine.

ECL, ASDF and cl-who
Now, where to put the Lisp http library cl-who? I copied the asd-file and the lisp-files to the ecl library folder and ran ecl as root:

kvaser@kvaser:~/lisp/cl-who-0.11.1$ sudo cp cl-who.asd /opt/ecl/lib/ecl-11.1.1/
kvaser@kvaser:~/lisp/cl-who-0.11.1$ sudo cp *.lisp /opt/ecl/lib/ecl-11.1.1/
kvaser@kvaser:~$ sudo /opt/ecl/bin/ecl
  ... ...
> (require 'asdf)

;;; Loading #P"/opt/ecl/lib/ecl-11.1.1/asdf.fas"
;;; Loading #P"/opt/ecl/lib/ecl-11.1.1/cmp.fas"
("ASDF" "CMP")

> (asdf:operate 'asdf:load-op :cl-who)    
  ... ...

Now, cl-who is compiled and installed, ready to use. Next time, it does not need to be compiled.

Hello LISP
I wrote a little Hello World program:

kvaser@kvaser:~/lisp$ cat hello.lisp 
(format T "Hello Lisp~%")
(quit)
kvaser@kvaser:~/lisp$ /opt/ecl/bin/ecl -load hello.lisp 
;;; Loading "/home/kvaser/lisp/hello.lisp"
Hello Lisp

Quite good (except I already know the file was loaded and it disturbs my output, but whatever. How about compiling it?


kvaser@kvaser:~/lisp$ /opt/ecl/bin/ecl -compile hello.lisp 
;;; Loading #P"/opt/ecl/lib/ecl-11.1.1/cmp.fas"
;;;
;;; Compiling hello.lisp.
;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=0
;;;
;;; End of Pass 1.
;;; Note:
;;;   Invoking external command:
;;;   gcc -I. -I/opt/ecl/include/ -I/opt/libffi/include -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -fPIC -Dlinux -O2 -w -c hello.c -o hello.o 
;;; Note:
;;;   Invoking external command:
;;;   gcc -o hello.fas -L/opt/ecl/lib/ /home/kvaser/lisp/hello.o -Wl,--rpath,/opt/ecl/lib/ -shared -L/opt/libffi/lib -L/opt/libffi/lib -lecl -lgmp -lgc -lffi -ldl -lm 
;;; Finished compiling hello.lisp.
;;;
kvaser@kvaser:~/lisp$ ls
cl-who-0.11.1  cl-who.tar.gz  hello.fas  hello.lisp
kvaser@kvaser:~/lisp$ ./hello.fas 
Segmentation fault
kvaser@kvaser:~/lisp$ /opt/ecl/bin/ecl -load hello.fas 
;;; Loading "/home/kvaser/lisp/hello.fas"
Hello Lisp

Ok, how to make a standalone executable?

> (compile-file "hello.lisp" :system-p t)
  ... ...
#P"/home/kvaser/lisp/hello.o"

> (c:build-program "hello" :lisp-files '("hello.o"))
  ... ...
#P"hello"
> (quit)
kvaser@kvaser:~/lisp$ ls
hello  hello.fas  hello.lisp  hello.o
kvaser@kvaser:~/lisp$ time ./hello
Hello Lisp

real	0m3.084s
user	0m2.920s
sys	0m0.160s
kvaser@kvaser:~/lisp$ time /opt/ecl/bin/ecl -load hello.fas 
;;; Loading "/home/kvaser/lisp/hello.fas"
Hello Lisp

real	0m3.127s
user	0m3.060s
sys	0m0.080s
kvaser@kvaser:~/lisp$ time /opt/ecl/bin/ecl -load hello.lisp
;;; Loading "/home/kvaser/lisp/hello.lisp"
Hello Lisp

real	0m3.113s
user	0m2.960s
sys	0m0.160s
kvaser@kvaser:~/lisp$ 

Clearly, some overhead is involved in invoking ECL. I compared to C:

kvaser@kvaser:~/lisp$ cat hello.c 
#include 

int main(int argc, char **argv) {
	printf("Hello C\n");
}
kvaser@kvaser:~/lisp$ gcc -o hello_c hello.c 
kvaser@kvaser:~/lisp$ time ./hello_c 
Hello C

real	0m0.012s
user	0m0.010s
sys	0m0.000s
kvaser@kvaser:~/lisp$ 

So, I can not use this method for CGI programming right away – each call to the webserver will take at least 3 seconds.