WangLu's Notes

Configuration files for different services are rarely independent. For example, in nftables, I might tag traffic with a firewall mark, and that mark is then used by systemd-networkd or in ip routes. Similarly, when the name of the primary network interface changes, multiple services like nftables, postfix, and samba need to be updated. Requirements I want to define core data in one place, then update all config files with a simple command. If a configuration file is modified by an external process (for example, a package update from a vendor or distribution), the changes must be handled gracefully. Either the merge should be automatic and permanent, or I should be notified to easily resolve any conflicts. It should be obvious within the config file itself what changes I have made. Existing Solutions I did some quick survey and found a few options. 1. Templates These tools render a template using provided data sources. To manage  /etc/config.txt , I would create a  /etc/config....

Now that I've settled on my VM image pipeline , the next logical step is to tackle networking. My Requirements So far, I've been using QEMU's default user-mode networking. It's convenient for quick tasks, allowing for easy port forwarding, Samba shares, and DNS with just a few flags. However, this setup is ultimately insufficient for my needs for a couple of key reasons: Security and Isolation: In the default user-mode setup, a VM can access the host's services via localhost . Worse, because it uses NAT, the VM can also access the host's entire LAN using the host's IP address. Ideally, VMs should have their own identifiable IP addresses, and more importantly, there should be strong network isolation between the host and the VMs. Centralized Auditing: I want to audit all network traffic from my VMs through a centralized solution. This means I need a way to route all VM traffic through a single point of control. Choosing the Right Tool For most people, tools...

I stumbled upon the Gentoo wiki page for systemd-nspawn , which in turn led me to nspawn.org , mkosi , and later systemd-sysupdate . mkosi quickly caught my eye because it's almost exactly what I wanted to build myself, as mentioned in a previous post . So, I decided to spend my "sysadmin fun quota" on it. Overview mkosi is similar to docker build or podman build , but it's designed for creating full OS images. It focuses on development and testing. For example, much like nix-shell , mkosi can quickly launch a sandboxed shell with a specific distribution and selected packages installed. The systemd project itself uses mkosi for testing across different distros. The re-introduction article  is a great read. Speed Note that this is by no means a rigid benchmark. My setup is an SSD with LUKS and an ext4 filesystem (without reflink support). Building Container Images mkosi is pretty fast. A simple mkosi command creates a fresh Debian image. I used the --incrementa...

Today, my pipeline regularly builds images for my disposable VMs . Here's the current process: A dedicated builder VM reads  Containerfile s for all VMs, including itself. The builder VM uses podman build to create container images for all VMs. The builder VM then uses bootc-image-builder to create disk images for all VMs. This process works well, but it has a significant issue: the disk images aren't built efficiently. Unlike container images, which benefit from reusable, cacheable layers, disk images are always built from scratch. This leads to long build times and limited opportunities for data deduplication. To address this, I've been exploring alternative options to improve the pipeline. Disk Image Formats and Deduplication My Current Format: QCOW2 I currently use QCOW2 with compression enabled. This format offers several features like snapshots, compression, and sparse files, which are useful when the underlying filesystem doesn't support them. However, if the f...

[Update] I managed to work out the AppArmor profile, and decided to go with guestmount for now. I am setting up a maintenance pipeline for my virtual machines . The pipeline has two main routines: The BACKUP routine: Every day, this routine shuts down each VM, backs up the data in /var, updates the VM's disk image if a new version is available, and then restarts it. The BUILD routine: Every week, this routine uses a special builder VM to create new disk images for all VMs. There is a scheduling conflict with the builder VM: the BACKUP routine needs to shut it down, while the BUILD routine needs it running. To resolve this, I merged both into a single set of systemd services that runs daily. The BUILD routine starts automatically when the builder VM starts, at the end of the BACKUP routine. The builder VM's systemd unit has an ExecCondition= property, which is skipped 6 days a week. Surprisingly, the most difficult part of this pipeline was not the scheduling, but the backup pro...

Today, various services (native, LXC, Docker) are running on my server. I'm mostly happy with the setup, but I decided to revisit my server's defenses under the assumption that a remote attacker or malicious code could compromise my services. A service might break out of its container or even gain root privilege. VMs are a better security boundary than containers; they can limit the damage if an attacker gains root privilege. I cannot afford to run a dedicated VM for each service, so I will need to carefully group the services and run a dedicated VM for each group. Each group should be carefully designed based on the data accessed and the features/capabilities required. For example, some VMs may have access to my photos, while others may not have network access. The Goal There are two particular issues I want to address: First, I want VM images to be easily reproducible, which makes backup and restore trivial. NixOS and GNU Guix System are great examples, where you only need to...

When I have important data on a device, I back it up to my server using dedicated user accounts. The other day, I checked /etc/passwd on my server and found entries like this: some-backup-user1:x:1003:1004:... some-backup-user2:x:1004:1007:... A few inconsistencies immediately bothered me: UID/GID Mismatches: Many users have UIDs that don't match their primary GIDs. While this technically works and might seem like just an aesthetic concern, I realized that UIDs and GIDs are crucial metadata. I need to preserve them accurately for future system migrations to maintain correct file ownership.ID Ambiguity: ID Ambiguity: The same number (e.g., 1004) could represent a User ID for one account and a Group ID for a completely different group. This overlap is a recipe for mistakes during administration tasks if I'm not paying close attention. Lack of Structure: User and group accounts created for very different purposes – regular logins, backup processes, container management, specifi...

My current server setup, based on Debian Stable and Docker, has served me reliably for years. It's stable, familiar, and gets the job done. However, an intriguing article I revisited recently about Fedora CoreOS, rpm-ostree, and OSTree native containers sparked my curiosity and sent me down a rabbit hole exploring alternative approaches to system management. Could there be a better way? Core Goals & Requirements Before diving into new technologies, I wanted to define what "better" means for my use case: The base operating system must update automatically and reliably. Hosted services (applications) should be updatable either automatically or manually, depending on the service. Configuration and data files need to be easy to modify, and crucially, automatically tracked and backed up. Current Setup: Debian Stable + Docker My current infrastructure consists of several servers, all running Debian Stable. System Updates are andled automatically via unattended-upgrades. Se...