WangLu's Notes

In Part 2 , I got a basic proof-of-concept running: a single headless microVM booting a minimal Debian kernel and forwarding Firefox back to my daily-driver desktop (“Viewer VM”) over VSOCK using Waypipe and PipeWire. It proved the concept, but it was essentially a static, single-instance test rig. To actually live in this setup like I used to in Qubes OS, I needed to turn this prototype into a scalable, multi-domain system. In this post, I’ll walk through the next iteration of the architecture: building a centralized VM lifecycle manager, isolating VMs under dedicated host users, scaling dynamic TAP networks, and running the Waypipe viewer inside a confined nested compositor container. 1. Domain Modeling & Host VM Management In Qubes OS, workloads are categorized into Disposable VMs (DispVMs) for ephemeral tasks and AppVMs for persistent user data, both derived from centralized Template VMs . While Qubes relies on Template VMs to maintain packages across domains, I chose ...

I decided to take the plunge into micro VMs. My goal? To set up a headless micro VM capable of running graphical programs remotely. As a first milestone, I wanted to get Firefox running and smoothly playing videos. (See Part 1 for a breakdown of why I passed on other isolation methods.) Overview At a high level, the concept is simple: I click an icon, and Firefox opens seamlessly on my screen while actually running securely in a VM. This setup is similar to disposable VMs in Qubes OS. When the program closes, the VM is destroyed, leaving absolutely zero trace on the disk. To pull this off, I needed to boot a micro VM with a minimal kernel and disk image, and seamlessly forward both graphics and audio to my daily-driver main VM. The Kernel Unlike standard VMs, micro VMs do not support PCI devices. Instead, they rely on different protocols (like virtio_mmio and virtio_blk ), which the kernel must support natively. Crucially, these drivers must be compiled directly into the kernel...

I’ve really been missing the experience of Qubes OS, where all programs are properly isolated. However, I can’t install it on my daily driver because I use that machine for gaming. Instead, I’ve been exploring ways to approximate that isolation on a standard Linux setup. Firefox Profiles Let’s start with the browser. There are many privacy and security benefits to separating profiles. My general workflow is as follows: I create separate .desktop files to run Firefox with different profiles. Each file uses a unique --name and StartupWMClass= . This prevents the icons from stacking together on the GNOME Panel. I use ImageMagick to tint the icons so I can tell the profiles apart at a glance: magick input.png -colorspace gray -fill "#cc0000" -tint 100 output.png The “new” Firefox profile manager didn’t work well for me. For example, I couldn’t set a default profile for opening external URLs. I eventually had to switch back to the old profile management style. However, whil...

Previously, I decided to set up a headless VM orchestrator for daily driving and gaming. In general, installing a minimal Debian and QEMU is fairly straightforward, but the devil is in the details. Here are my notes on the process. See also: part 1 Overall Setup The machine is a laptop. The built-in display is connected to the iGPU, and an external display is connected to the dGPU. I prefer to use only the external display; the built-in one is rarely used. I have two VMs: one as a daily driver and the other for gaming. Both have GPU passthrough. Only one runs at a time. Resources Reserved for the Host CPU: 1 core, 2 threads. RAM: 2GB (typical usage is around 500MB). Disk: 32GB. The OS only needs ~2GB, but I need extra space for cache, log, OS images, etc. CPU The main QEMU process and IO thread should be pinned to the host CPUs. vCPU threads should be pinned to the CPUs reserved for the VM. CPU pinning cannot be done through QEMU command-line arguments. I had to talk to the QEM...

For years, I've been playing games and doing everything else on the same PC. Hoever, after a recent positive experience with Qubes OS, I no longer feel comfortable keeping my games, passwords, and other stuff all in the same place. Because of this, I decided to change my gaming setup. My requirements are: Performance. Gaming should be functional and playable. Isolation. At a minimum, game binaries should not have access to my other files. Quick Switch. I need to be able to switch between gaming and non-gaming easily. Single Display. All my workflows should be performance well on a single display. I do have a secondary display, but I want to rarely use it. So, here goes my adeventure. Option 1: Dual Boot This is the classic solution, which sounds boring but almost always works. Performance: Good. All operating systems have direct access to physical hardware. Isolation: OK-ish, each OS technically has access to all data from other OSes. Disk encryption and Secure Boot can mitigate th...

I've been running Qubes OS on Machine A alongside Windows for a while. My setup involved storing the unencrypted  /boot  partition and the LUKS header on an external USB drive. Recently, I planned to install Qubes on Machine B, also in a dual-boot configuration. However, the complexity jumped significantly: Machine B has Secure Boot enabled because BitLocker requires it. On previous installs, I grew tired of toggling Secure Boot in the BIOS every time I switched operating systems. I only have one USB drive. Managing separate  /boot  partitions for two different Qubes installations on a single thumb drive is messy. After some experimentation, I found a way to solve both problems. Sharing One USB Drive for Multiple Qubes Installations The solution is elegant: Don't store  /boot  on the USB drive. Instead, move  /boot  to the encrypted internal disk partition. The USB drive's only job is to unlock that partition and hand over control to the system. O...

I've been running a few containers for several years. They were all running under rootless Docker with a single user. Initially, I planned to  migrate the containers to VMs , but I couldn't get a stable workflow after about two months of effort. Later,  gVisor caught my attention , and I decided to migrate to Podman with gVisor instead. The new plan is to run each container with  --userns=auto  and use Quadlet for systemd integration. This approach provides better isolation and makes writing firewall rules easier. I'm now close to migrating all my containers. Here are a couple of rough edges I'd like to share. Network Layout I compared  various networking options  and spent a few hours trying the one-interface-per-group approach before giving up. I settled on a single macvlan network and decided to use static IP addresses for my containers. To prevent a randomly assigned IP address from conflicting with a predefined one, I allocated a large IP range for my ...

My original plan was to stabilize my  VM pipeline  before deploying containers using a hardened stack of Podman, QEMU, SELinux, and user namespaces ( --userns=auto ). However, the pipeline's complexity grew, requiring script rewrites and schema redesigns, and the process took much longer than anticipated. In the meantime, an interesting alternative has captured my attention:  gVisor . It occupies a unique space between traditional SELinux policies and full-blown virtual machines, offering a compelling set of trade-offs. What is gVisor? At its core, gVisor is an application kernel, written in the memory-safe language Go, that provides an additional layer of isolation between containerized applications and the host operating system. It's essentially a user-space implementation of the Linux kernel's system call interface. The security model is explained  here . gVisor in Practice gVisor provides an OCI-compliant runtime called  runsc , which can be almost transpare...

Now that I've settled on my VM image pipeline , the next logical step is to tackle networking. My Requirements So far, I've been using QEMU's default user-mode networking. It's convenient for quick tasks, allowing for easy port forwarding, Samba shares, and DNS with just a few flags. However, this setup is ultimately insufficient for my needs for a couple of key reasons: Security and Isolation: In the default user-mode setup, a VM can access the host's services via localhost . Worse, because it uses NAT, the VM can also access the host's entire LAN using the host's IP address. Ideally, VMs should have their own identifiable IP addresses, and more importantly, there should be strong network isolation between the host and the VMs. Centralized Auditing: I want to audit all network traffic from my VMs through a centralized solution. This means I need a way to route all VM traffic through a single point of control. Choosing the Right Tool For most people, tools...

The central question is: how do you manage secrets like SSH keys, API keys, and passwords for disposable VMs ? 🤷‍♂️ Let's establish some ground rules for this scenario. Suppose I want to pass an API key to the VM  chimera , which is run by the  chimera-runner user on the host. My security requirements are: On the host, only root and  chimera-runner  should have access to the secrets. In VM  chimera , only root and relevant service users should have access to the secrets. No one from other VMs, including their root users, should have access to VM  chimera 's secrets. The guest VMs themselves are not trusted . The bootc documentation on this topic is very informative. On a high level, there are a few ways to achieve this. 1. OEM Strings / Firmware QEMU can pass data to a VM via SMBIOS OEM strings ( -smbios ) or firmware configuration ( -fw_cfg ). Notably, both methods are supported by systemd-creds using special keys. This approach is practical for sma...

Just learned about a few interesting and useful stuff, when playing with bootc: systemd Components systemd-tmpfiles  and systemd-sysusers  allows managing files and users in a declarative way. Originally I learned about this for building bootc images, but later I realized that they are also very useful on Debian. I learned systemd-credential  as a way of passing ssh authorized keys to a QEMU VM, but after reading more, I realized it can be used in other interesting ways. My favorite one is with LoadCredential=, I can run a script with DynamicUser=yes and the script can access some root-only secrets. I finally decided to migrate from cron to systemd-timer. systemd-timer is more interesting and handy than expected, and the migration process is less painful than expected. SELinux Actually I heared about SELinux many years ago. Over the time I just know SELinux as "something about security, similar but more complicated to AppArmor". Recently I got to learn more about it: - ht...

A few days ago, while browsing security topics online, Qubes OS surfaced—whether via YouTube recommendations or search results, I can't recall precisely. Intrigued by its unique approach to security through compartmentalization, I delved into the documentation and watched some demos. My interest was piqued enough that I felt compelled to install it and give it a try firsthand. My overall first impression of Qubes OS is highly positive. Had I discovered it earlier, I might have reconsidered starting my hardware password manager project. Conceptually, Qubes OS is not much different from running a bunch of virtual machines simultaneously. However, its brilliance lies in the seamless desktop integration and the well-designed template system, making it far more user-friendly than a manual VM setup. I was particularly impressed by the concept of disposable VMs for temporary tasks and the clear separation of critical functions like networking (sys-net) and USB handling (sys-usb) into the...