Skip to main content

mkosi: First Impressions

I stumbled upon the Gentoo wiki page for systemd-nspawn, which in turn led me to nspawn.org, mkosi, and later systemd-sysupdate.

mkosi quickly caught my eye because it's almost exactly what I wanted to build myself, as mentioned in a previous post. So, I decided to spend my "sysadmin fun quota" on it.

Overview

mkosi is similar to docker build or podman build, but it's designed for creating full OS images. It focuses on development and testing. For example, much like nix-shell, mkosi can quickly launch a sandboxed shell with a specific distribution and selected packages installed. The systemd project itself uses mkosi for testing across different distros.

The re-introduction article is a great read.

Speed

Note that this is by no means a rigid benchmark.

My setup is an SSD with LUKS and an ext4 filesystem (without reflink support).

Building Container Images

mkosi is pretty fast. A simple mkosi command creates a fresh Debian image. I used the --incremental flag for subsequent builds.

  • First run: ~30s

  • Second run (after trivial changes): ~5s

Using mkosi -p systemd allows the container to boot (via systemd-nspawn -b), which adds only a few seconds to the build time.

Building VM Images

Building a VM image with mkosi --include mkosi-vm is a bit slower, likely due to the extra steps for installing a bootloader and kernel.

  • First run: ~1m 30s

  • Second run (after minor changes): ~30s

Comparison with bootc

I tried to build a fresh CentOS image using both tools.

mkosi --include mkosi-vm -d centos

  • Duration: ~1m 30s

  • Output disk size: 1.2 GB

podman pull quay.io/centos-bootc/bootc-image-builder:latest && \
podman run ... \
    quay.io/centos-bootc/bootc-image-builder:latest \
    --type raw ... \
    quay.io/centos-bootc/centos-bootc:stream9

  • Duration: ~4m 30s

  • Output disk size: 1.9 GB

Notes:

  • The bootc-image-builder was pre-pulled, and this time isn't included in the measurement.
  • The time to pull the base CentOS image is included. 
  • I'm generating a raw image here instead of QCOW2.

Again, these numbers aren't directly comparable outside of my specific setup.

  • bootc-image-builder runs in a VM, while mkosi runs directly on the host.

  • centos and centos-bootc are different distributions, and their configurations (like installed packages) are also very different. This is obvious from the difference in their final image sizes.

Running Images with systemd-nspawn

I attempted to get unprivileged systemd-nspawn working but failed:

  • systemd-nsresourced.socket and systemd-mountfsd.socket must be running.

  • systemd-mountfsd complains that the image is untrusted unless it's signed or located in a trusted location.

  • I got stuck on another error.

Eventually, I resorted to using sudo systemd-nspawn -U, which worked well. The -b flag "boots" the image by running systemd/init as PID 1.

Running Images with QEMU

mkosi --kvm vm works nicely.

Notes:

Observations, Thoughts and Concerns

  • mkosi is deeply integrated with systemd. Its configuration files are also following the systemd style: e.g. declarative, ini, drop-in overrides.

  • I wasn't able to test the performance benefits of reflink, because my filesystem doesn't support it and the disk images were small anyway.

  • I also wasn't able to test if SELinux works. Supposedly, it needs an extra flag in mkosi.conf and might be slow. On the other hand, it works out-of-the-box in bootc images.

  • I don't really miss Containerfiles much. I usually just need to copy files, and for my use case, a Containerfile would essentially just be running my scripts with bind mounts. Plus, I don't use many layers. But I might miss having an immutable Linux setup.

  • mkosi supports many popular distributions. while bootc only support Fedora/CentOS.

  • mkosi may add surprising modifications to the image:

  • zvol doesn't seem very reliable, so I'll probably avoid using it for another few years.

Conclusion

mkosi is a very interesting tool. While I'm not ready to migrate my entire image-building pipeline yet, I might consider replacing my current LXC setup with it.

Labels:

Comments

Post a Comment

Popular posts from this blog

Determine Perspective Lines With Off-page Vanishing Point

In perspective drawing, a vanishing point represents a group of parallel lines, in other words, a direction. For any point on the paper, if we want a line towards the same direction (in the 3d space), we simply draw a line through it and the vanishing point. But sometimes the vanishing point is too far away, such that it is outside the paper/canvas. In this example, we have a point P and two perspective lines L1 and L2. The vanishing point VP is naturally the intersection of L1 and L2. The task is to draw a line through P and VP, without having VP on the paper. I am aware of a few traditional solutions: 1. Use extra pieces of paper such that we can extend L1 and L2 until we see VP. 2. Draw everything in a smaller scale, such that we can see both P and VP on the paper. Draw the line and scale everything back. 3. Draw a perspective grid using the Brewer Method. #1 and #2 might be quite practical. #3 may not guarantee a solution, unless we can measure distances/p...
3 comments
Read More »

Exploring Immutable Distros and Declarative Management

My current server setup, based on Debian Stable and Docker, has served me reliably for years. It's stable, familiar, and gets the job done. However, an intriguing article I revisited recently about Fedora CoreOS, rpm-ostree, and OSTree native containers sparked my curiosity and sent me down a rabbit hole exploring alternative approaches to system management. Could there be a better way? Core Goals & Requirements Before diving into new technologies, I wanted to define what "better" means for my use case: The base operating system must update automatically and reliably. Hosted services (applications) should be updatable either automatically or manually, depending on the service. Configuration and data files need to be easy to modify, and crucially, automatically tracked and backed up. Current Setup: Debian Stable + Docker My current infrastructure consists of several servers, all running Debian Stable. System Updates are andled automatically via unattended-upgrades. Se...
Post a Comment
Read More »

Qubes OS: First Impressions

A few days ago, while browsing security topics online, Qubes OS surfaced—whether via YouTube recommendations or search results, I can't recall precisely. Intrigued by its unique approach to security through compartmentalization, I delved into the documentation and watched some demos. My interest was piqued enough that I felt compelled to install it and give it a try firsthand. My overall first impression of Qubes OS is highly positive. Had I discovered it earlier, I might have reconsidered starting my hardware password manager project. Conceptually, Qubes OS is not much different from running a bunch of virtual machines simultaneously. However, its brilliance lies in the seamless desktop integration and the well-designed template system, making it far more user-friendly than a manual VM setup. I was particularly impressed by the concept of disposable VMs for temporary tasks and the clear separation of critical functions like networking (sys-net) and USB handling (sys-usb) into the...
Post a Comment
Read More »