Linux Daily Driver Setup Part 3: VM Control Panel

Now that I have my VMs running, what if I want to switch between them? What if I want to put the VM or the host to sleep, or shut them down?

In this post, I’ll discuss a few options.

Just a quick overview of my setup:

• Only one VM is running at a time.
• All VMs have GPU passthrough, and the GPU is connected to a display.
• I have a secondary display, which I prefer not to use unless absolutely necessary.

See also:

• Part 1
• Part 2

Option 1: Deep Integration with Guest OS

Ideally, I could just click on a “Power off and Switch to VM X” menu from within the guest OS.

This isn’t difficult to support on the host side: the guest OS could pass the VM name to the host (e.g., via a serial port), and the host would automatically start the next VM when the current one exits.

However, I didn’t find an easy way to implement this menu, especially considering I plan to support multiple operating systems. I also couldn’t find a way to prevent the guest from shutting down without providing the next VM’s name.

It sounds like it would require a lot of custom scripting, so I decided to pass on this idea.

Option 2: GPU Rebinding + TTY Menu

I figured the logic had to be implemented on the host side. The idea here is that the host reclaims the GPU when a VM exits.

This requires two steps:

1. Rebind the GPU from vfio-pci to a “real” driver after a VM exits.
2. Rebind the GPU from the “real” driver back to vfio-pci before a VM starts.

Step 1 is generally easy because vfio-pci is pretty lightweight, but Step 2 is not, especially for nouveau. I also needed to make sure to disconnect all possible GPU usages (e.g., fbcon), otherwise the driver would likely hang.

In the end, I couldn’t make it stable enough, and I definitely didn’t want to risk hanging the host. Time for a new option.

Option 3: Menu in VM

Since I prefer to stick with the vfio-pci driver, a natural idea was to launch a dedicated VM just for the menu.

The kernel and rootfs could be heavily stripped down to do nothing but show a menu and pass the user’s choice back to the host. It’s a bit of work, but doable.

In practice, however, a minimal VM took about 6 seconds to boot. I managed to pinpoint the bottleneck: GPU initialization. The VM boots much faster without it.

Ultimately, I couldn’t find a good solution. My only finding was that OVMF/UEFI is required to initialize the GPU; without it, the GPU won’t work and the kernel might just hang (the RIP register doesn’t change). Using a dumped or downloaded ROM file didn’t help in my case.

I didn’t bother implementing the menu because the boot delay was just too slow for practical use.

Option 4: Web Server

Another idea was to implement a web server with a simple UI, allowing me to control the host using my phone.

I don’t think it would be difficult to build something that just works, but making it secure enough would be a challenge. Plus, I don’t like that it requires a second device.

Option 5: TTY Menu in Secondary Monitor

Running out of ideas for reusing the primary monitor, I decided to compromise and use the secondary display.

It was straightforward to implement:

• The menu is built using whiptail.
• Mask the default getty@tty2 service and run my menu service on TTY2.
• Use chvt to switch terminals, and setfont to set a huge font size.
• Write to /sys/module/kernel/parameters/consoleblank to make the screen turn off automatically.
• The screen turns on automatically when a VM stops, probably due to keyboard/mouse events after evdev passthrough switching.
• I also added a few options to the menu like “sleep” and “power off”, so I can control the host without having to log in.

In practice, this works really well. It just requires that secondary display.

Thoughts

While Option 5 ended up being the best compromise, I really wish Option 2 or 3 had worked better. That way, the entire setup would work on a single display. Maybe there is a better solution for GPU initialization out there. I’ll probably revisit this later.