WangLu's Notes

I recently tried to create a simple systemd service to send an email notification, but my initial approach with mail and sendmail failed with a strange permission error. My original service file looked like this: [Service] ExecStart=mail --subject=Subject recipient@example.com The error message was a bit of a head-scratcher: warning: mail_queue_enter: create file maildrop/....: Permission denied . A quick search pointed me to the cause : the postdrop binary has setgid. However, the systemd setting NoNewPrivileges=true prevents this. While I hadn't explicitly used that setting, I was using DynamicUser=true , which implies and enforces NoNewPrivileges=true . This meant my service, running as a temporary user, couldn't get the permissions it needed to interact with the mail queue. Note that this implication cannot be disabled/overriden. I wanted to avoid creating a new, dedicated user for this task. I realized that the problem was how mail and sendmail directly interact wit...

I stumbled upon the Gentoo wiki page for systemd-nspawn , which in turn led me to nspawn.org , mkosi , and later systemd-sysupdate . mkosi quickly caught my eye because it's almost exactly what I wanted to build myself, as mentioned in a previous post . So, I decided to spend my "sysadmin fun quota" on it. Overview mkosi is similar to docker build or podman build , but it's designed for creating full OS images. It focuses on development and testing. For example, much like nix-shell , mkosi can quickly launch a sandboxed shell with a specific distribution and selected packages installed. The systemd project itself uses mkosi for testing across different distros. The re-introduction article  is a great read. Speed Note that this is by no means a rigid benchmark. My setup is an SSD with LUKS and an ext4 filesystem (without reflink support). Building Container Images mkosi is pretty fast. A simple mkosi command creates a fresh Debian image. I used the --incrementa...

Today, my pipeline regularly builds images for my disposable VMs . Here's the current process: A dedicated builder VM reads  Containerfile s for all VMs, including itself. The builder VM uses podman build to create container images for all VMs. The builder VM then uses bootc-image-builder to create disk images for all VMs. This process works well, but it has a significant issue: the disk images aren't built efficiently. Unlike container images, which benefit from reusable, cacheable layers, disk images are always built from scratch. This leads to long build times and limited opportunities for data deduplication. To address this, I've been exploring alternative options to improve the pipeline. Disk Image Formats and Deduplication My Current Format: QCOW2 I currently use QCOW2 with compression enabled. This format offers several features like snapshots, compression, and sparse files, which are useful when the underlying filesystem doesn't support them. However, if the f...

The central question is: how do you manage secrets like SSH keys, API keys, and passwords for disposable VMs ? 🤷‍♂️ Let's establish some ground rules for this scenario. Suppose I want to pass an API key to the VM  chimera , which is run by the  chimera-runner user on the host. My security requirements are: On the host, only root and  chimera-runner  should have access to the secrets. In VM  chimera , only root and relevant service users should have access to the secrets. No one from other VMs, including their root users, should have access to VM  chimera 's secrets. The guest VMs themselves are not trusted . The bootc documentation on this topic is very informative. On a high level, there are a few ways to achieve this. 1. OEM Strings / Firmware QEMU can pass data to a VM via SMBIOS OEM strings ( -smbios ) or firmware configuration ( -fw_cfg ). Notably, both methods are supported by systemd-creds using special keys. This approach is practical for sma...

[Update] I managed to work out the AppArmor profile, and decided to go with guestmount for now. I am setting up a maintenance pipeline for my virtual machines . The pipeline has two main routines: The BACKUP routine: Every day, this routine shuts down each VM, backs up the data in /var, updates the VM's disk image if a new version is available, and then restarts it. The BUILD routine: Every week, this routine uses a special builder VM to create new disk images for all VMs. There is a scheduling conflict with the builder VM: the BACKUP routine needs to shut it down, while the BUILD routine needs it running. To resolve this, I merged both into a single set of systemd services that runs daily. The BUILD routine starts automatically when the builder VM starts, at the end of the BACKUP routine. The builder VM's systemd unit has an ExecCondition= property, which is skipped 6 days a week. Surprisingly, the most difficult part of this pipeline was not the scheduling, but the backup pro...

Just learned about GNU Stow , which is a tool for managing symlink farm. Basically the idea is to store all files in one place, then create symlink all around the system pointing to your files. There are various use cases, like dot files and installing/uninstalling packages. But I mostly use it for tracking system config files, similar to how NixOS works. In fact I wrote my own scripts with "cp -rs", but GNU Stow works much better.