I stumbled upon the Gentoo wiki page for systemd-nspawn
, which in turn led me to nspawn.org, mkosi
, and later systemd-sysupdate
.
mkosi
quickly caught my eye because it's almost exactly what I wanted to build myself, as
Overview
mkosi
is similar to docker build
or podman build
, but it's designed for creating full OS images. It focuses on development and testing. For example, much like nix-shell
, mkosi
can quickly launch a sandboxed shell with a specific distribution and selected packages installed. The mkosi
for testing across different distros.
The
Speed
Note that this is by no means a rigid benchmark.
My setup is an SSD with LUKS and an ext4 filesystem (without reflink support).
Building Container Images
mkosi
is pretty fast. A simple mkosi
command creates a fresh Debian image. I used the --incremental
flag for subsequent builds.
First run: ~30s
Second run (after trivial changes): ~5s
Using mkosi -p systemd
allows the container to boot (via systemd-nspawn -b
), which adds only a few seconds to the build time.
Building VM Images
Building a VM image with mkosi --include mkosi-vm
is a bit slower, likely due to the extra steps for installing a bootloader and kernel.
First run: ~1m 30s
Second run (after minor changes): ~30s
Comparison with bootc
I tried to build a fresh CentOS image using both tools.
mkosi --include mkosi-vm -d centos
Duration: ~1m 30s
Output disk size: 1.2 GB
podman pull quay.io/centos-bootc/bootc-image-builder:latest && \
podman run ... \
quay.io/centos-bootc/bootc-image-builder:latest \
--type raw ... \
quay.io/centos-bootc/centos-bootc:stream9
Duration: ~4m 30s
Output disk size: 1.9 GB
Notes:
- The
bootc-image-builder
was pre-pulled, and this time isn't included in the measurement. - The time to pull the base CentOS image is included.
- I'm generating a raw image here instead of QCOW2.
Again, these numbers aren't directly comparable outside of my specific setup.
bootc-image-builder
runs in a VM, whilemkosi
runs directly on the host.centos
andcentos-bootc
are different distributions, and their configurations (like installed packages) are also very different. This is obvious from the difference in their final image sizes.
Running Images with systemd-nspawn
I attempted to get unprivileged systemd-nspawn
working but failed:
systemd-nsresourced.socket
andsystemd-mountfsd.socket
must be running.systemd-mountfsd
complains that the image is untrusted unless it's signed or located in a .trusted location I got stuck on
.another error
Eventually, I resorted to using sudo systemd-nspawn -U
, which worked well. The -b
flag "boots" the image by running systemd/init
as PID 1.
Running Images with QEMU
mkosi --kvm vm
works nicely.
Notes:
Credentials are
visible in the command-line arguments I'm not a fan of all the default flags for QEMU, but
mkosi
provides many options for customization.
Observations, Thoughts and Concerns
mkosi
is deeply integrated with systemd. Its configuration files are also following the systemd style: e.g. declarative, ini, drop-in overrides.I wasn't able to test the performance benefits of reflink, because my filesystem doesn't support it and the disk images were small anyway.
I also wasn't able to test if SELinux works. Supposedly, it needs an extra flag in mkosi.conf and might be slow. On the other hand, it works out-of-the-box in
bootc
images.I don't really miss
Containerfile
s much. I usually just need to copy files, and for my use case, aContainerfile
would essentially just be running my scripts with bind mounts. Plus, I don't use many layers. But I might miss having an immutable Linux setup.mkosi
supports many popular distributions. whilebootc
only support Fedora/CentOS.mkosi
may add surprising modifications to the image:mkosi
. It actually used to depend on it, but that dependency wasdoesn't use debootstrap
. Not sure if this approach is hacky.removed mkosi
may .inject its own SSH server unit zvol
doesn't seem very reliable, so I'll probably avoid using it for another few years.
Conclusion
mkosi
is a very interesting tool. While I'm not ready to migrate my entire image-building pipeline yet, I might consider replacing my current LXC setup with it.
Comments