Recently I've been modernizing the build infrastructure at work, introducing containerization technology. I've intentionally left out Docker as a possible work horse because our build server is behind an insanely picky security gateway that would go mad every time the former would reach out to download images.

My aim with containerization was to provide a build environment that is

  1. isolated (from the build host) to avoid undetected build dependencies,
  2. independent enough (from the build host) so that we can build x86 software without polluting the x86_64 host with multiarch packages as well as build with ancient tooling even when our build host runs the latest Ubuntu version and finally
  3. invariant so that we can rely on a build environment that will not change (without our intention).

At first I thought of LXC but that still seemed a bit overkill to me. The only thing that would run inside those containers is the build process itself so the container would not even need to be booted. What I needed was rather something like chroot on steroids...

Enter systemd-nspawn!

Creating a rootfs for the Container

I'm using debootstrap for this as all our build environments are Debian based. Each of these rootfs directories gets its own BTRFS subvolume so we can efficiently snapshot them (see below).

Example:

$ btrfs subvol create /var/lib/machines/deb10-amd64-crossbuild
$ debootstrap \
    --arch=i386 --components=main,universe --variant=minbase \
    --include=systemd-container,git,cmake,build-essential \
    buster /var/lib/machines/deb10-amd64-crossbuild \
    https://deb.debian.org/debian

Launching Stuff

In order to execute a process inside a container (or boot the container) there is systemd-nspawn. Depending on the desired nature of the container there are lots of options to tweak the behaviour, from completely volatile systems to persistent environments or snapshots. I highly recommend diving into the man page.

The following demonstrates what I have chosen to run our build scripts:

$ systemd-nspawn \
    --directory=/var/lib/machines/deb10-i386-unittest \
    --personality=x86 --ephemeral --read-only \
    --bind-ro=/etc/passwd --bind-ro=/etc/group \
    --bind-ro=/opt --bind=/home \
    -u developer1 /path/to/build-script.sh

The container is operated in ephemeral read-only mode, i.e. a read-only snapshot of the actual rootfs is taken for execution and deleted right after the process inside the container has terminated.

This has several benefits:

  • the system part of the container is entirely read-only, contributing to our invariance goal,
  • multiple build jobs can make use of the 'same' container (rootfs) without interfering as they actually end up running on volatile snapshots of the rootfs and have a UUID in their name,
  • snapshot creation is very cheap as we're using BTRFS subvolumes
  • bind-mounting /etc/passwd, /etc/group and /home in combination with -u we can provide a user environment almost identical to the one on the host, no need for complicated artifact extraction after the build before the container is destroyed.

A Wrapper for the Unprivileged

At this point there is only one issue left to be solved: unfortunately systemd-nspawn cannot be run as regular user, even when the container is set to utilize user namespaces:

Invoking containers without privileges is not supported by nspawn, and this is unlikely to change, as I fail to see any strong usecase for this...

— Lennart Poettering via systemd-devel

Our build jobs run as unprivileged users as a matter of course, and apart from that I also wanted to enable our developers to spawn an interactive build environment via shell so they could work on issues without the indirection of a CI server in between.

So I decided to write a wrapper script for systemd-nspawn that enforces safe operation and make it executable via sudo for all relevant users. This way the only change that needed to be done to our build jobs in order to make the jump to containerized builds was to place the wrapper before the actual build script invocation:

# before
$ /home/ci/path/to/build-script.sh

# after
$ nspawn-run -n deb10-i386-unittest /home/ci/path/to/build-script.sh

Developers can seamlessly enter their own private container environment for working with archaic tooling or interactive cross-building:

user@buildhost:~$ nspawn-run -n deb7-i386-veryold
Spawning container deb7-i386-veryold-d05e0226f83a8241 on /var/lib/machines/.#machine.deb7-i386-veryolde227f05cfe94f92f.
Press ^] three times within 1s to kill container.
user@deb7-i386-veryold-d05e0226f83a8241:~$
  (...)
user@deb7-i386-veryold-d05e0226f83a8241:~$ exit
exit
Container deb7-i386-veryold-d05e0226f83a8241 exited successfully.