WangLu's Notes

Configuration files for different services are rarely independent. For example, in nftables, I might tag traffic with a firewall mark, and that mark is then used by systemd-networkd or in ip routes. Similarly, when the name of the primary network interface changes, multiple services like nftables, postfix, and samba need to be updated. Requirements I want to define core data in one place, then update all config files with a simple command. If a configuration file is modified by an external process (for example, a package update from a vendor or distribution), the changes must be handled gracefully. Either the merge should be automatic and permanent, or I should be notified to easily resolve any conflicts. It should be obvious within the config file itself what changes I have made. Existing Solutions I did some quick survey and found a few options. 1. Templates These tools render a template using provided data sources. To manage  /etc/config.txt , I would create a  /etc/config....

Now that I've settled on my VM image pipeline , the next logical step is to tackle networking. My Requirements So far, I've been using QEMU's default user-mode networking. It's convenient for quick tasks, allowing for easy port forwarding, Samba shares, and DNS with just a few flags. However, this setup is ultimately insufficient for my needs for a couple of key reasons: Security and Isolation: In the default user-mode setup, a VM can access the host's services via localhost . Worse, because it uses NAT, the VM can also access the host's entire LAN using the host's IP address. Ideally, VMs should have their own identifiable IP addresses, and more importantly, there should be strong network isolation between the host and the VMs. Centralized Auditing: I want to audit all network traffic from my VMs through a centralized solution. This means I need a way to route all VM traffic through a single point of control. Choosing the Right Tool For most people, tools...

I recently tried to create a simple systemd service to send an email notification, but my initial approach with mail and sendmail failed with a strange permission error. My original service file looked like this: [Service] ExecStart=mail --subject=Subject recipient@example.com The error message was a bit of a head-scratcher: warning: mail_queue_enter: create file maildrop/....: Permission denied . A quick search pointed me to the cause : the postdrop binary has setgid. However, the systemd setting NoNewPrivileges=true prevents this. While I hadn't explicitly used that setting, I was using DynamicUser=true , which implies and enforces NoNewPrivileges=true . This meant my service, running as a temporary user, couldn't get the permissions it needed to interact with the mail queue. Note that this implication cannot be disabled/overriden. I wanted to avoid creating a new, dedicated user for this task. I realized that the problem was how mail and sendmail directly interact wit...

I stumbled upon the Gentoo wiki page for systemd-nspawn , which in turn led me to nspawn.org , mkosi , and later systemd-sysupdate . mkosi quickly caught my eye because it's almost exactly what I wanted to build myself, as mentioned in a previous post . So, I decided to spend my "sysadmin fun quota" on it. Overview mkosi is similar to docker build or podman build , but it's designed for creating full OS images. It focuses on development and testing. For example, much like nix-shell , mkosi can quickly launch a sandboxed shell with a specific distribution and selected packages installed. The systemd project itself uses mkosi for testing across different distros. The re-introduction article  is a great read. Speed Note that this is by no means a rigid benchmark. My setup is an SSD with LUKS and an ext4 filesystem (without reflink support). Building Container Images mkosi is pretty fast. A simple mkosi command creates a fresh Debian image. I used the --incrementa...

Today, my pipeline regularly builds images for my disposable VMs . Here's the current process: A dedicated builder VM reads  Containerfile s for all VMs, including itself. The builder VM uses podman build to create container images for all VMs. The builder VM then uses bootc-image-builder to create disk images for all VMs. This process works well, but it has a significant issue: the disk images aren't built efficiently. Unlike container images, which benefit from reusable, cacheable layers, disk images are always built from scratch. This leads to long build times and limited opportunities for data deduplication. To address this, I've been exploring alternative options to improve the pipeline. Disk Image Formats and Deduplication My Current Format: QCOW2 I currently use QCOW2 with compression enabled. This format offers several features like snapshots, compression, and sparse files, which are useful when the underlying filesystem doesn't support them. However, if the f...

The central question is: how do you manage secrets like SSH keys, API keys, and passwords for disposable VMs ? 🤷‍♂️ Let's establish some ground rules for this scenario. Suppose I want to pass an API key to the VM  chimera , which is run by the  chimera-runner user on the host. My security requirements are: On the host, only root and  chimera-runner  should have access to the secrets. In VM  chimera , only root and relevant service users should have access to the secrets. No one from other VMs, including their root users, should have access to VM  chimera 's secrets. The guest VMs themselves are not trusted . The bootc documentation on this topic is very informative. On a high level, there are a few ways to achieve this. 1. OEM Strings / Firmware QEMU can pass data to a VM via SMBIOS OEM strings ( -smbios ) or firmware configuration ( -fw_cfg ). Notably, both methods are supported by systemd-creds using special keys. This approach is practical for sma...

[Update] I managed to work out the AppArmor profile, and decided to go with guestmount for now. I am setting up a maintenance pipeline for my virtual machines . The pipeline has two main routines: The BACKUP routine: Every day, this routine shuts down each VM, backs up the data in /var, updates the VM's disk image if a new version is available, and then restarts it. The BUILD routine: Every week, this routine uses a special builder VM to create new disk images for all VMs. There is a scheduling conflict with the builder VM: the BACKUP routine needs to shut it down, while the BUILD routine needs it running. To resolve this, I merged both into a single set of systemd services that runs daily. The BUILD routine starts automatically when the builder VM starts, at the end of the BACKUP routine. The builder VM's systemd unit has an ExecCondition= property, which is skipped 6 days a week. Surprisingly, the most difficult part of this pipeline was not the scheduling, but the backup pro...