My Experience with libvirt

My daily driver setup has been working exceptionally well. If you missed them, you can check out the previous posts for details:

• Part 1
• Part 2
• Part 3

Recently, I decided to harden my setup by writing custom AppArmor profiles and nftables rules. During my research, libvirt kept popping up in tutorials. In fact, most “QEMU/KVM” guides simply assume you are using it.

Initially, I decided against using libvirt because I wasn’t a fan of its design philosophy. However, I kept hearing that it can automatically generate AppArmor profiles and firewall rules for each VM. Furthermore, this article highlighted several things that libvirt genuinely simplifies. Intrigued, I decided to dive in and get some first-hand experience.

The plan was to migrate my existing VM setup (which relies on bare QEMU scripts) to libvirt to see if it was a good fit. I specifically wanted to evaluate the parts of libvirt I was previously skeptical about:

• Reliance on a highly privileged daemon.
• Configuration stored in managed XML files, which makes it harder to create reusable components or templates.

The Good Parts

• As noted in this article, while bare QEMU/KVM lacks a stable API, libvirt provides a very stable XML API.
• CPU pinning and lifecycle management (like gracefully shutting down a VM) are as simple as just a few lines of XML. With bare QEMU, I had to write a Python daemon to parse the QMP protocol just to achieve this. libvirt also seamlessly handles a lot of edge cases during shutdowns.

The OK Parts

• domxml-from-native didn’t work for me; this might be a Debian-specific issue. Thankfully, it wasn’t too difficult to manually recreate the XML file from my existing QEMU arguments.
• Instead of using virsh define, I found I could use virsh create. This reads from an XML config and creates a transient VM. I love this approach because it lets me maintain control over my XML configs, making it possible to use scripts or Jinja templates to define reusable templates.
• The highly privileged daemon is a mixed bag. Sometimes root access is required to configure network interfaces or AppArmor profiles, but I generally prefer to isolate those operations. For example, I’d rather pre-configure the network interfaces and define AppArmor profiles in systemd service files to keep the daemon rootless. That said, it’s not a dealbreaker:
  • The Arch Wiki notes that the daemon isn’t strictly required in all scenarios.
  • I had already ended up writing my own custom daemons for CPU pinning and lifecycle management anyway.
• Networking has its quirks. libvirt can add a managed TAP device, but out of the box, it lacks NAT and network filtering.
  • It can use my existing TAP device as an unmanaged interface, but then it won’t apply network filters.
  • I couldn’t seem to manually fix the name of the TAP device, which makes writing custom nftables rules a headache.
  • Network configurations have to be defined separately (though they can be created transiently).
  • While libvirt can auto-generate network filters, any custom logic requires a separate, permanent XML config. I’d much rather just write plain nftables rules directly.
• libvirt automatically creates AppArmor profiles for each VM, restricting the VM to only read its own disk image. The profile libvirt provides is actually quite reusable. I’ve used it in the past for server projects, even if it’s not perfect.

The Bad Parts

• libvirt pulls in a lot of dependencies. Most annoyingly, nwfilter has a hard dependency on iptables. I’ve heard libvirt supports nftables now, so this might just be another Debian-specific quirk.
• It wasn’t easy to assign a unique user to each VM. Setting the DAC seclabel resulted in a permission error regarding “master-key.aes”. This is likely because /var/lib/libvirt/qemu is restricted exclusively to the libvirt-qemu user.
• I couldn’t get sound to work at all (I use ALSA). I probably just needed to add the libvirt-qemu user to the right audio groups, but it was another hurdle.

Conclusion

It is undeniable that libvirt is highly scalable and offers a rock-solid API. However, for my specific use case, it makes 80% of the setup incredibly easy while making the last 20% frustratingly difficult. To use it efficiently, you really have to commit to speaking the “libvirt language”.

I could probably iron out the remaining kinks if I spent a few more hours on it, but I don’t see the point right now. I’ll likely reconsider it in the future when I start experimenting with disposable VMs.