VM Setup Part 2: QEMU

Previously, I decided to set up a headless VM orchestrator for daily driving and gaming.

In general, installing a minimal Debian and QEMU is fairly straightforward, but the devil is in the details. Here are my notes on the process.

Overall Setup

• The machine is a laptop. The built-in display is connected to the iGPU, and an external display is connected to the dGPU.
• I prefer to use only the external display; the built-in one is rarely used.
• I have two VMs: one as a daily driver and the other for gaming.
• Both have GPU passthrough.
• Only one runs at a time.

Resources Reserved for the Host

• CPU: 1 core, 2 threads.
• RAM: 2GB (typical usage is around 500MB).
• Disk: 32GB. The OS only needs ~2GB, but I need extra space for cache, log, OS images, etc.

CPU

• The main QEMU process and IO thread should be pinned to the host CPUs.
• vCPU threads should be pinned to the CPUs reserved for the VM.
• CPU pinning cannot be done through QEMU command-line arguments. I had to talk to the QEMU process via QMP (though this would be easier with libvirt).

RAM

• Enable 1GB huge pages and reserve them so the host cannot use them.
• Instruct QEMU to use all reserved huge pages, pre-allocate all needed memory, and avoid using memory from other sources.

GPU Passthrough

• Various guides are available online:
  • Arch Wiki
  • Debian Wiki
• Kernel parameters for vfio_pci didn’t work for me.
  • The GPU was still bound to some other driver by default. I probably needed to adjust the module order in initramfs.
  • This was fine in my case because the TTY is bound to the iGPU, and the dGPU is not used by default. It was easy to dynamically unbind the default driver and bind it to vfio-pci.
• The display remained blank unless I added OVMF UEFI firmware.
  • Interestingly, this was very tricky to debug: there was no display and no terminal output (maybe a serial port would have helped?).
  • Luckily, I had tried QEMU with a GUI before this headless setup, which was easier to debug since error messages appeared on the virtual display.
  • It seems the GPU needs to run the VBIOS, triggered by UEFI, to initialize.
  • I couldn’t make it work using a manual ROM file (dumped or downloaded) without UEFI.
• It takes 4–6 seconds for a VM to initialize the GPU during boot. I haven’t been able to eliminate this delay.

Network

• A standard bridge didn’t work because the host uses Wi-Fi.
• MACVTAP/MACVLAN didn’t work because the VM needs to communicate with the host’s network.
• I decided to use TAP + IP routing (NAT), which was easy to set up with systemd-networkd.
  • Debian still uses ifupdown + dhcpcd for wireless connections.
  • systemd-networkd didn’t always propagate the DNS server to the VM via DHCP.
  • This was likely because my wireless connection isn’t perfectly stable, so I had to specify the DNS value manually.

USB

• I chose to forward specific USB devices instead of the host controller, since the host also needs the keyboard.
• Contrary to what some docs mention, USB hot-plugging seems to work by default. QEMU correctly forwards devices even if I unplug and replug them on the fly.
• Keyboard & Mouse stutter
  • evdev passthrough:
    • Helps with the stutter a little bit, but doesn’t eliminate it entirely.
    • Works well for devices in /dev/input/by-id/.
    • Does not support hot-plugging.
    • Caps Lock didn’t work.
    • Media keys (e.g., volume up/down) didn’t work.
  • Switching from PS/2 to virtio seemed to help.
  • Documentation on this is a bit lacking, but AI tools were helpful.
• USB game controllers
  • The controller stopped working after the VM woke from sleep.
  • It turned out to be also related to keyboard stutter.
  • Physically reconnecting the controller fixes it. I can probably write a script to dynamically reconnect it later.

Audio

• I don’t have PipeWire or PulseAudio installed on the host.
• I chose to use ALSA.
  • The default arguments didn’t work; I had to check /proc/asound/devices to get the correct card number.
  • Need to escape commas in the QEMU command line (e.g., -audiodev alsa,out.dev=hw:1,,0).

Bluetooth

• Passing the adapter via USB didn’t work.
  • It seemed to cause slow boot times on Bazzite.
  • I might try this solution later.

VM Isolation

• Each VM runs under its own unprivileged user.
  • I needed to grant access to vfio, USB, and audio devices. This was easy using sysusers.d.
  • This isolation wasn’t as easy when I tried libvirt, but I probably just didn’t dig deep enough.
• AppArmor
  • libvirt’s template is quite reusable. I also used it in my previous VM setup.
• Firewall
  • Currently, I cannot distinguish between the two VMs in the firewall rules because they share the same TAP device. I’ll have to use separate ones eventually.

Lifecycle Management

• Gracefully shutting down a VM from the host
  • QMP’s system_powerdown sends an ACPI shutdown signal, which didn’t work (Bazzite triggers sleep by default).
  • The QEMU Guest Agent’s guest-shutdown is more reliable (I need to set up the agent inside the guest, obviously).
  • Neither command guarantees an immediate shutdown; they just enqueue the operation. I still need to wait for the QEMU process to actually stop.
  • It is possible to implement this with a hacky, minimal shell script by blindly sending messages to the QMP/guest agent without parsing the output. Much easier with libvirt.
• Suspend/Sleep
  • When I put a VM to sleep, I want the host to sleep as well.
  • To do that, I wrote a Python script to parse events from the QMP socket and watch for the SUSPEND event. (libvirt already handles this; I just need to register a script).
  • systemctl suspend can suspend the host, but again, when the command finishes, it doesn’t guarantee the suspend (and resume) cycle is complete. It simply enqueues the operation.
  • This means I can’t immediately wake up the VM via script right after this command.
  • The proper way is to register sleep hooks with systemd.
  • Alternatively, do nothing: after waking up the host with the keyboard, I just need to press a few more keys to wake up the VM.

OS

• I chose Fedora Silverblue for my daily driver.
  • Booting is fast.
  • After investigating how Flatpak works, I really like it; it feels similar to Android’s app isolation.
• I chose Bazzite for gaming.
  • It just works out of the box. I didn’t need a single terminal command to play games (the only exception was when I enabled Secure Boot).
  • Around 80% of my game library is supported, which is great. However, this means I might still have to set up a Windows VM later for the remaining games.

Conclusion

This setup has been working well for a while now. It was a nice way to get myself updated on the modern Linux desktop and GPU passthrough experience.

I like that this setup borrows great concepts from both Qubes OS and Proxmox. It’s interesting to note the shift in my approach: in my previous VM setup, I wanted to optimize the VMs by giving them as few resources as possible. Now, I’m doing the exact opposite.

I also worked on a few other things that I might discuss in upcoming posts:

• Trying to make it easier to switch/launch VMs without logging into the host.
• Getting more interested in libvirt and deciding to give it a try.
• Plans to “Qubes-ify” my setup (e.g., setting up disposable VMs).