Recently I've been modernizing the build infrastructure at work, introducing containerization technology. I've intentionally left out Docker as a possible work horse because our build server is behind an insanely picky security gateway that would go mad every time the former would reach out to download images.
My aim with containerization was to provide a build environment that is
- isolated (from the build host) to avoid undetected build dependencies,
- independent enough (from the build host) so that we can build x86 software without polluting the x86_64 host with multiarch packages as well as build with ancient tooling even when our build host runs the latest Ubuntu version and finally
- invariant so that we can rely on a build environment that will not change (without our intention).
At first I thought of LXC but that still seemed a bit overkill to me. The only thing that would run inside those containers is the build process itself so the container would not even need to be booted. What I needed was rather something like chroot on steroids...
Creating a rootfs for the Container
debootstrap for this as all our build environments are
Debian based. Each of these rootfs directories gets
its own BTRFS subvolume so we can efficiently snapshot them (see below).
$ btrfs subvol create /var/lib/machines/deb10-amd64-crossbuild $ debootstrap \ --arch=i386 --components=main,universe --variant=minbase \ --include=systemd-container,git,cmake,build-essential \ buster /var/lib/machines/deb10-amd64-crossbuild \ https://deb.debian.org/debian
In order to execute a process inside a container (or boot the container) there
systemd-nspawn. Depending on the desired nature of the container there are
lots of options to tweak the behaviour, from completely volatile systems to
persistent environments or snapshots. I highly recommend diving into the man
The following demonstrates what I have chosen to run our build scripts:
$ systemd-nspawn \ --directory=/var/lib/machines/deb10-i386-unittest \ --personality=x86 --ephemeral --read-only \ --bind-ro=/etc/passwd --bind-ro=/etc/group \ --bind-ro=/opt --bind=/home \ -u developer1 /path/to/build-script.sh
The container is operated in ephemeral read-only mode, i.e. a read-only snapshot of the actual rootfs is taken for execution and deleted right after the process inside the container has terminated.
This has several benefits:
- the system part of the container is entirely read-only, contributing to our invariance goal,
- multiple build jobs can make use of the 'same' container (rootfs) without interfering as they actually end up running on volatile snapshots of the rootfs and have a UUID in their name,
- snapshot creation is very cheap as we're using BTRFS subvolumes
/homein combination with
-uwe can provide a user environment almost identical to the one on the host, no need for complicated artifact extraction after the build before the container is destroyed.
Wrapper for the Unprivileged
At this point there is only one issue left to be solved: unfortunately
systemd-nspawn cannot be run as regular user, even when the container is set
to utilize user
Invoking containers without privileges is not supported by nspawn, and this is unlikely to change, as I fail to see any strong usecase for this...
- — Lennart Poettering via systemd-devel
Our build jobs run as unprivileged users as a matter of course, and apart from that I also wanted to enable our developers to spawn an interactive build environment via shell so they could work on issues without the indirection of a CI server in between.
So I decided to write a wrapper
systemd-nspawn that enforces safe operation and make it executable via
for all relevant users. This way the only change that needed to be done to our
build jobs in order to make the jump to containerized builds was to place the
wrapper before the actual build script invocation:
# before $ /home/ci/path/to/build-script.sh # after $ nspawn-run -n deb10-i386-unittest /home/ci/path/to/build-script.sh
Developers can seamlessly enter their own private container environment for working with archaic tooling or interactive cross-building:
user@buildhost:~$ nspawn-run -n deb7-i386-veryold Spawning container deb7-i386-veryold-d05e0226f83a8241 on /var/lib/machines/.#machine.deb7-i386-veryolde227f05cfe94f92f. Press ^] three times within 1s to kill container. user@deb7-i386-veryold-d05e0226f83a8241:~$ (...) user@deb7-i386-veryold-d05e0226f83a8241:~$ exit exit Container deb7-i386-veryold-d05e0226f83a8241 exited successfully.