Qubesify My Daily Driver Part 3: VM Management, Nested Compositors, and Containment

In Part 2, I got a basic proof-of-concept running: a single headless microVM booting a minimal Debian kernel and forwarding Firefox back to my daily-driver desktop (“Viewer VM”) over VSOCK using Waypipe and PipeWire.

It proved the concept, but it was essentially a static, single-instance test rig. To actually live in this setup like I used to in Qubes OS, I needed to turn this prototype into a scalable, multi-domain system.

In this post, I’ll walk through the next iteration of the architecture: building a centralized VM lifecycle manager, isolating VMs under dedicated host users, scaling dynamic TAP networks, and running the Waypipe viewer inside a confined nested compositor container.

---

1. Domain Modeling & Host VM Management

In Qubes OS, workloads are categorized into Disposable VMs (DispVMs) for ephemeral tasks and AppVMs for persistent user data, both derived from centralized Template VMs.

While Qubes relies on Template VMs to maintain packages across domains, I chose not to use that model. Instead, I construct the base rootfs disk image myself via automated image builds. This provides a cleaner, fully reproducible, and declarative base system without the overhead of template sync states.

To organize domains, I set up a configuration file that acts as the single source of truth:

• Base Configurations & Inheritance: Rather than defining every VM from scratch, the configuration defines shared templates (specifying baseline vCPU counts, memory, and base disk images). Specific domain definitions inherit from a base template and override targeted properties like custom border colors or extra memory.
• Ephemeral Workloads: Disposable VMs run as ephemeral instances booting from a shared read-only base image with all runtime writes held in a guest RAM overlay. Once the application exits, the VM shuts down and all runtime state vanishes.
• Persistent AppVMs (Planned): Persistent AppVMs with dedicated storage attachments (such as separate /home volumes) are mapped out in the configuration schema, planned for future iterations.

To orchestrate microVM lifecycles without running monolithic daemon processes, I wrote a lightweight controller script (vm-manager).

Hypervisor-Level User Isolation

Running all QEMU processes under root or a single shared user defeats the purpose of compartmentalization.

• Each active VM is dynamically assigned a Context ID (CID) from a designated range (e.g., 2000 to 3000).
• Each CID maps directly to an isolated host user (e.g., CID 2042 runs under microvm-user-2042, UID 2042).
• VM resources and advisory file locks are coordinated dynamically during startup and teardown, guaranteeing strict hypervisor-level privilege separation between running domains.

Dynamic TAP Devices

Network interfaces are provisioned on demand rather than statically allocated. A templated systemd service handles the creation and destruction of per-VM TAP devices (e.g., tap-vm-2042), automatically cleaning up unused interfaces on shutdown to avoid cluttering the host network stack.

---

2. Containerized Nested Compositor in the Viewer VM

On the viewer desktop, untrusted Waypipe rendering streams should not connect directly to the primary host Wayland compositor. Instead, incoming streams launch an unprivileged, rootless container running a nested compositor via systemd socket activation.

Bridging these components uncovered several low-level plumbing quirks.

Passing Metadata via Stream Headers

When an application launches inside a microVM, the host forwards the stream to the viewer desktop. To apply domain-specific styling (such as window borders and tags) without maintaining out-of-band control channels, the connection broker injects a compact metadata header before the raw payload. This header prepends the domain’s verified CID, name, and color code directly to the stream before handing off the socket.

Podman Socket Activation & Colon Parsing

With systemd Accept=yes sockets, systemd generates instance strings derived from network socket pairs (e.g., vsock:2000:12345-vsock:2:23456). Colons in these instance strings broke Podman container name validation (Podman Issue #22874).

To resolve this, the systemd template invokes podman run --rm to let Podman auto-generate valid container names.

Staging the Container Entrypoint: The Colon Path Trap

The container entrypoint is a custom Python script that orchestrates the viewer session: it reads the metadata header from the incoming socket, generates the runtime Sway config, launches Sway, and bridges the incoming VSOCK stream to Sway’s Wayland socket.

To avoid rebuilding the container image on every code tweak during development, I decided to bind-mount the script directly into the container. However, mounting it straight from my home directory wasn’t ideal because it would require giving the container read access to my home folder.

Systemd offers an elegant directive for this: LoadCredential=, which securely stages a specified file into the service’s runtime credentials directory before execution. But the colon-in-instance-string problem struck again: systemd names the credential path after the instance string (e.g., /run/credentials/service@vsock:2000:.../), and Podman’s --volume / -v flag parser choked on the colon in the source path.

In the end, I worked around this by setting up a dedicated pre-launch service that copies the script to a predictable, colon-free shared directory (/run/microvm/), allowing the containerized Waypipe service to bind-mount it cleanly.

The EBUSY Wayland Socket Collision

When launching the nested compositor inside the container, it initially crashed on startup with:

Unable to open Wayland socket: Device or resource busy

The nested compositor acts simultaneously as a Wayland client (talking to the host) and a Wayland server (listening for the guest application). When mounting the host display directly to /tmp/wayland-0, libwayland-server detected an existing socket, assumed it was a stale artifact from a crashed session, and attempted to unlink() it. Because the socket was an active bind mount, the unlink() syscall failed with EBUSY.

This issue is also related to not mounting the Wayland lock file into the container. While mounting the lock file might resolve the conflict, doing so exposes unnecessary host state that is better avoided.

The cleaner fix was path isolation: bind-mounting the host compositor socket to /tmp/host-wayland-0 and exporting WAYLAND_DISPLAY=host-wayland-0. This leaves the standard /tmp/wayland-0 path free for the nested compositor’s internal server socket.

Bridging to Waypipe with os.splice()

To tie the socket forwarding together with minimal latency, I use a lightweight Python bridge script utilizing os.splice() to ferry raw byte streams directly between the incoming VSOCK descriptor and Waypipe’s local UNIX socket. The container launcher reads the initial metadata header, configures the environment, and triggers Waypipe in listening mode.

---

3. Visual Security Cues, Sway, and Rendering

With the display pipeline running, the next objective was enforcing visual isolation cues: distinct domain border colors and non-spoofable window titles.

The Move from Cage to Sway

I initially tested cage, a lightweight Wayland kiosk compositor. However, cage lacks native support for configurable titlebars, borders, or runtime styling, and trying to force decorations through it hit a dead end.

I switched to running sway in nested kiosk mode. With a dynamically generated runtime configuration, Sway provides declarative styling out of the box:

default_border pixel 4
client.focused #e63946 #e63946 #ffffff #e63946 #e63946
for_window [all] border pixel 4
for_window [all] fullscreen disable
title_format "[untrusted] %title"
exec waypipe --socket /tmp/waypipe-stream.sock client

Window Dragging Quirks

One subtle behavior with nested Sway is window movement: clicking and dragging the window titlebar directly does not move the container window because mouse events are captured and routed straight into the inner nested compositor.

However, using Sway’s modifier shortcut (Super + drag) moves the nested window as expected. While it might be possible to intercept bare titlebar drags, Super + drag works reliably and avoids brittle event-hooking hacks.

Graphics Acceleration

I initially explored enabling hardware acceleration using the host’s integrated GPU (iGPU). However, the microVM setup does not expose the host iGPU directly to the guest environment. For lightweight tools and terminal sessions, software rendering remains responsive enough, though I plan to investigate iGPU passthrough or NVIDIA container runtimes for GPU-heavy workloads later on.

---

Side Note: “Vibe Coding” Low-Level Linux Plumbing

Working with systemd socket activation, SELinux constraints, compositor behaviors, and nested Wayland interactions involves wrestling with countless low-level edge cases.

Pair programming with an AI assistant to analyze error logs, trace syscall quirks, and evaluate architectural trade-offs sped up this plumbing work significantly—functioning like an extra set of eyes when debugging odd low-level interactions.

---

What’s Next?

With dynamic VM management and nested compositor containers working, the roadmap ahead includes:

1. Persistent AppVM Storage: Finalizing persistent volume attachments and clean single-instance guards for stateful domains.
2. Qubes-Style Explicit Clipboard Isolation: Replacing automatic clipboard sync with an explicit global keybinding (Super+Shift+C / Super+Shift+V) to prevent background microVMs from snooping on clipboard data.
3. Host-to-Guest Input Method (IME): Forwarding Fcitx5/IBus input contexts seamlessly into sandboxed guest apps.

Stay tuned for Part 4!