An Adventure with Qubes OS

I've been experimenting with Qubes OS on my new laptop and wanted to share some notes on the experience.

Hardware

Overall, Qubes OS works quite well on my hardware. Aside from typical issues like deep sleep, speaker performance, and touchpad scroll speed, the experience has been smooth. I particularly like that I can boot directly from a microSD card. This allowed me to move the /boot partition to the card while completely disabling USB access in dom0 for better security.

Detached /boot and LUKS Header

Moving /boot and the LUKS header to a microSD card is a fun project, but it has some drawbacks:

• I have to remember to mount /boot before updating dom0.
• The system won't shut down properly if I forget to unmount /boot.

Testing Qubes OS 4.3 rc3

I decided to test the Qubes OS 4.3 rc3 release by performing an in-place upgrade. Unfortunately, the system failed to boot afterward.

dracut Issues

After the upgrade, the system would hang before prompting me for my LUKS password. Eventually, the watchdog timer would kick in and drop me into an emergency shell.

Using the emergency shell and the installation media, I was able to investigate. I realized something was wrong with dracut, as it seemed unable to detect the encrypted disk. I tried including more files in the dracut configuration based on information from various sources, but that didn't help. So I gave up. Thankfully, the upgrade tool created an LVM snapshot, which made it easy to revert the changes. I did have to manually downgrade the kernel and Xen using the installation media to get my Xen domains working again.

After more research, I found a bug (1, 2) in the version of dracut included in Qubes OS 4.3. Essentially, the crypt module stops working when systemd is available. I also believe the systemd-cryptsetup module wasn't automatically included because the LUKS header was on a separate device, leading dracut to assume that LUKS decryption wasn't needed.

The fix was simple: manually enable the systemd-cryptsetup module in the dracut configuration.

amdgpu Issues

After the upgrade, only the old kernel worked, that's how I could fix the dracut issue. With the new kernels, the system would boot to a blank screen. Removing the rhgb and quiet kernel parameters revealed some log messages, but the blank screen remained.

Adding the nomodeset parameter disabled the graphics driver, which allowed me to enter the LUKS password and log in. However, this caused Xorg and LightDM to fail, with Xorg repeatedly crashing.

It was clear this was an issue with the amdgpu module. I tried several kernel parameters without success:

• amdgpu.modeset=1
• amdgpu.dc=0
• amdgpu.dpm=0
• amdgpu.ppfeaturemask=0xffffb

Eventually, I found that amdgpu.dcdebugmask=0x10 resolved the problem.

qubesd Issues

After finally booting into the system, I couldn't attach any block devices, including my /boot partition, to dom0. This turned out to be a bug in qubesd, which I reported.

Backup Strategy

Backing up data in Qubes OS can be tricky due to its design. Here is the high-level strategy I've been planning:

• Create as few templates as possible.
• Only use Salt to configure templates. So I only need to back up Salt files in dom0.
• Changes to dom0 are either managed by Salt, or the relevant files are included in the backup. Examples
  • Xfce settings.
  • Qube settings/features, /etc/qubes
  • /etc/default/grub
  • /etc/dracut.conf.d

My backup process is as follows:

• I use a dedicated disposable VM with restic installed. The actual backup scripts and SSH keys remain in dom0.
• Data from dom0 and other qubes is archived using tar. Use --transform to prepend the VM name to the path.
• Use qvm-run --pass-io --no-gui to pass the scripts, SSH keys an data to the disposable VM, which then runs the scripts to execute the backup.

There are a few things to consider:

• Using --pass-io could have security implications. It might be possible to mitigate this by limiting the number of bytes passed and saving output to a file.
• The tar archives are currently extracted in the disposable VM. If the data isn't trusted, this could be a security risk. In such cases, the extraction step could be skipped.
• Accessing the backed-up data requires starting a VM. An alternative could be to create an LVM snapshot and back that up directly.

Thought: Manging Qubes like Containers

There are many similarities between Qubes VMS and containers (docker, podman):

• Template VMs are similar to building containers.
• AppVMs are like running containers with persistent volumes.
• Disposable VMs are like running containers without persistent volumes.

There are some gaps:

• One container can be based on others, the shared part are often stored only once as layers. While cloning a template VM can use reflink for initial efficienty, changes will evetually cause the data to diverge. bootc might help bridge this gap.
• Containers support bind mounts, which is very convenient for backup. While virtiofsd could work for Xen, but I guess there are security concerns. An alternative could be to centralize all data in one qube and share it with others over the network, but again, there could be security concerns.

Conclusion

It's been a fun and challenging journey exploring Qubes OS. While there were some hurdles with hardware and system upgrades, working through them has been a valuable learning experience. The security architecture is powerful, and I'm excited to continue finding new ways to make it work for my setup.