WangLu's Notes

I decided to take the plunge into micro VMs. My goal? To set up a headless micro VM capable of running graphical programs remotely. As a first milestone, I wanted to get Firefox running and smoothly playing videos. (See Part 1 for a breakdown of why I passed on other isolation methods.) Overview At a high level, the concept is simple: I click an icon, and Firefox opens seamlessly on my screen while actually running securely in a VM. This setup is similar to disposable VMs in Qubes OS. When the program closes, the VM is destroyed, leaving absolutely zero trace on the disk. To pull this off, I needed to boot a micro VM with a minimal kernel and disk image, and seamlessly forward both graphics and audio to my daily-driver main VM. The Kernel Unlike standard VMs, micro VMs do not support PCI devices. Instead, they rely on different protocols (like virtio_mmio and virtio_blk ), which the kernel must support natively. Crucially, these drivers must be compiled directly into the kernel...

I’ve really been missing the experience of Qubes OS, where all programs are properly isolated. However, I can’t install it on my daily driver because I use that machine for gaming. Instead, I’ve been exploring ways to approximate that isolation on a standard Linux setup. Firefox Profiles Let’s start with the browser. There are many privacy and security benefits to separating profiles. My general workflow is as follows: I create separate .desktop files to run Firefox with different profiles. Each file uses a unique --name and StartupWMClass= . This prevents the icons from stacking together on the GNOME Panel. I use ImageMagick to tint the icons so I can tell the profiles apart at a glance: magick input.png -colorspace gray -fill "#cc0000" -tint 100 output.png The “new” Firefox profile manager didn’t work well for me. For example, I couldn’t set a default profile for opening external URLs. I eventually had to switch back to the old profile management style. However, whil...

Now that I have my VMs running, what if I want to switch between them? What if I want to put the VM or the host to sleep, or shut them down? In this post, I’ll discuss a few options. Just a quick overview of my setup: Only one VM is running at a time. All VMs have GPU passthrough, and the GPU is connected to a display. I have a secondary display, which I prefer not to use unless absolutely necessary. See also: Part 1 Part 2 Option 1: Deep Integration with Guest OS Ideally, I could just click on a “Power off and Switch to VM X” menu from within the guest OS. This isn’t difficult to support on the host side: the guest OS could pass the VM name to the host (e.g., via a serial port), and the host would automatically start the next VM when the current one exits. However, I didn’t find an easy way to implement this menu, especially considering I plan to support multiple operating systems. I also couldn’t find a way to prevent the guest from shutting down without providing the next V...

Previously, I decided to set up a headless VM orchestrator for daily driving and gaming. In general, installing a minimal Debian and QEMU is fairly straightforward, but the devil is in the details. Here are my notes on the process. See also: part 1 Overall Setup The machine is a laptop. The built-in display is connected to the iGPU, and an external display is connected to the dGPU. I prefer to use only the external display; the built-in one is rarely used. I have two VMs: one as a daily driver and the other for gaming. Both have GPU passthrough. Only one runs at a time. Resources Reserved for the Host CPU: 1 core, 2 threads. RAM: 2GB (typical usage is around 500MB). Disk: 32GB. The OS only needs ~2GB, but I need extra space for cache, log, OS images, etc. CPU The main QEMU process and IO thread should be pinned to the host CPUs. vCPU threads should be pinned to the CPUs reserved for the VM. CPU pinning cannot be done through QEMU command-line arguments. I had to talk to the QEM...

For years, I've been playing games and doing everything else on the same PC. Hoever, after a recent positive experience with Qubes OS, I no longer feel comfortable keeping my games, passwords, and other stuff all in the same place. Because of this, I decided to change my gaming setup. My requirements are: Performance. Gaming should be functional and playable. Isolation. At a minimum, game binaries should not have access to my other files. Quick Switch. I need to be able to switch between gaming and non-gaming easily. Single Display. All my workflows should be performance well on a single display. I do have a secondary display, but I want to rarely use it. So, here goes my adeventure. Option 1: Dual Boot This is the classic solution, which sounds boring but almost always works. Performance: Good. All operating systems have direct access to physical hardware. Isolation: OK-ish, each OS technically has access to all data from other OSes. Disk encryption and Secure Boot can mitigate th...

I've been running Qubes OS on Machine A alongside Windows for a while. My setup involved storing the unencrypted  /boot  partition and the LUKS header on an external USB drive. Recently, I planned to install Qubes on Machine B, also in a dual-boot configuration. However, the complexity jumped significantly: Machine B has Secure Boot enabled because BitLocker requires it. On previous installs, I grew tired of toggling Secure Boot in the BIOS every time I switched operating systems. I only have one USB drive. Managing separate  /boot  partitions for two different Qubes installations on a single thumb drive is messy. After some experimentation, I found a way to solve both problems. Sharing One USB Drive for Multiple Qubes Installations The solution is elegant: Don't store  /boot  on the USB drive. Instead, move  /boot  to the encrypted internal disk partition. The USB drive's only job is to unlock that partition and hand over control to the system. O...

I've been running a few containers for several years. They were all running under rootless Docker with a single user. Initially, I planned to  migrate the containers to VMs , but I couldn't get a stable workflow after about two months of effort. Later,  gVisor caught my attention , and I decided to migrate to Podman with gVisor instead. The new plan is to run each container with  --userns=auto  and use Quadlet for systemd integration. This approach provides better isolation and makes writing firewall rules easier. I'm now close to migrating all my containers. Here are a couple of rough edges I'd like to share. Network Layout I compared  various networking options  and spent a few hours trying the one-interface-per-group approach before giving up. I settled on a single macvlan network and decided to use static IP addresses for my containers. To prevent a randomly assigned IP address from conflicting with a predefined one, I allocated a large IP range for my ...

Now that I've settled on my VM image pipeline , the next logical step is to tackle networking. My Requirements So far, I've been using QEMU's default user-mode networking. It's convenient for quick tasks, allowing for easy port forwarding, Samba shares, and DNS with just a few flags. However, this setup is ultimately insufficient for my needs for a couple of key reasons: Security and Isolation: In the default user-mode setup, a VM can access the host's services via localhost . Worse, because it uses NAT, the VM can also access the host's entire LAN using the host's IP address. Ideally, VMs should have their own identifiable IP addresses, and more importantly, there should be strong network isolation between the host and the VMs. Centralized Auditing: I want to audit all network traffic from my VMs through a centralized solution. This means I need a way to route all VM traffic through a single point of control. Choosing the Right Tool For most people, tools...

Today, my pipeline regularly builds images for my disposable VMs . Here's the current process: A dedicated builder VM reads  Containerfile s for all VMs, including itself. The builder VM uses podman build to create container images for all VMs. The builder VM then uses bootc-image-builder to create disk images for all VMs. This process works well, but it has a significant issue: the disk images aren't built efficiently. Unlike container images, which benefit from reusable, cacheable layers, disk images are always built from scratch. This leads to long build times and limited opportunities for data deduplication. To address this, I've been exploring alternative options to improve the pipeline. Disk Image Formats and Deduplication My Current Format: QCOW2 I currently use QCOW2 with compression enabled. This format offers several features like snapshots, compression, and sparse files, which are useful when the underlying filesystem doesn't support them. However, if the f...

The central question is: how do you manage secrets like SSH keys, API keys, and passwords for disposable VMs ? 🤷‍♂️ Let's establish some ground rules for this scenario. Suppose I want to pass an API key to the VM  chimera , which is run by the  chimera-runner user on the host. My security requirements are: On the host, only root and  chimera-runner  should have access to the secrets. In VM  chimera , only root and relevant service users should have access to the secrets. No one from other VMs, including their root users, should have access to VM  chimera 's secrets. The guest VMs themselves are not trusted . The bootc documentation on this topic is very informative. On a high level, there are a few ways to achieve this. 1. OEM Strings / Firmware QEMU can pass data to a VM via SMBIOS OEM strings ( -smbios ) or firmware configuration ( -fw_cfg ). Notably, both methods are supported by systemd-creds using special keys. This approach is practical for sma...

[Update] I managed to work out the AppArmor profile, and decided to go with guestmount for now. I am setting up a maintenance pipeline for my virtual machines . The pipeline has two main routines: The BACKUP routine: Every day, this routine shuts down each VM, backs up the data in /var, updates the VM's disk image if a new version is available, and then restarts it. The BUILD routine: Every week, this routine uses a special builder VM to create new disk images for all VMs. There is a scheduling conflict with the builder VM: the BACKUP routine needs to shut it down, while the BUILD routine needs it running. To resolve this, I merged both into a single set of systemd services that runs daily. The BUILD routine starts automatically when the builder VM starts, at the end of the BACKUP routine. The builder VM's systemd unit has an ExecCondition= property, which is skipped 6 days a week. Surprisingly, the most difficult part of this pipeline was not the scheduling, but the backup pro...

Today, various services (native, LXC, Docker) are running on my server. I'm mostly happy with the setup, but I decided to revisit my server's defenses under the assumption that a remote attacker or malicious code could compromise my services. A service might break out of its container or even gain root privilege. VMs are a better security boundary than containers; they can limit the damage if an attacker gains root privilege. I cannot afford to run a dedicated VM for each service, so I will need to carefully group the services and run a dedicated VM for each group. Each group should be carefully designed based on the data accessed and the features/capabilities required. For example, some VMs may have access to my photos, while others may not have network access. The Goal There are two particular issues I want to address: First, I want VM images to be easily reproducible, which makes backup and restore trivial. NixOS and GNU Guix System are great examples, where you only need to...

Just learned about a few interesting and useful stuff, when playing with bootc: systemd Components systemd-tmpfiles  and systemd-sysusers  allows managing files and users in a declarative way. Originally I learned about this for building bootc images, but later I realized that they are also very useful on Debian. I learned systemd-credential  as a way of passing ssh authorized keys to a QEMU VM, but after reading more, I realized it can be used in other interesting ways. My favorite one is with LoadCredential=, I can run a script with DynamicUser=yes and the script can access some root-only secrets. I finally decided to migrate from cron to systemd-timer. systemd-timer is more interesting and handy than expected, and the migration process is less painful than expected. SELinux Actually I heared about SELinux many years ago. Over the time I just know SELinux as "something about security, similar but more complicated to AppArmor". Recently I got to learn more about it: - ht...