Skip to main content

An Adventure with Qubes OS

I've been experimenting with Qubes OS on my new laptop and wanted to share some notes on the experience.

Hardware

Overall, Qubes OS works quite well on my hardware. Aside from typical issues like deep sleep, speaker performance, and touchpad scroll speed, the experience has been smooth. I particularly like that I can boot directly from a microSD card. This allowed me to move the /boot partition to the card while completely disabling USB access in dom0 for better security.

Detached /boot and LUKS Header

Moving /boot and the LUKS header to a microSD card is a fun project, but it has some drawbacks:

  • I have to remember to mount /boot before updating dom0.
  • The system won't shut down properly if I forget to unmount /boot.

Testing Qubes OS 4.3 rc3

I decided to test the Qubes OS 4.3 rc3 release by performing an in-place upgrade. Unfortunately, the system failed to boot afterward.

dracut Issues

After the upgrade, the system would hang before prompting me for my LUKS password. Eventually, the watchdog timer would kick in and drop me into an emergency shell.

Using the emergency shell and the installation media, I was able to investigate. I realized something was wrong with dracut, as it seemed unable to detect the encrypted disk. I tried including more files in the dracut configuration based on information from various sources, but that didn't help. So I gave up. Thankfully, the upgrade tool created an LVM snapshot, which made it easy to revert the changes. I did have to manually downgrade the kernel and Xen using the installation media to get my Xen domains working again.

After more research, I found a bug (1, 2) in the version of dracut included in Qubes OS 4.3. Essentially, the crypt module stops working when systemd is available. I also believe the systemd-cryptsetup module wasn't automatically included because the LUKS header was on a separate device, leading dracut to assume that LUKS decryption wasn't needed.

The fix was simple: manually enable the systemd-cryptsetup module in the dracut configuration.

amdgpu Issues

After the upgrade, only the old kernel worked, that's how I could fix the dracut issue. With the new kernels, the system would boot to a blank screen. Removing the rhgb and quiet kernel parameters revealed some log messages, but the blank screen remained.

Adding the nomodeset parameter disabled the graphics driver, which allowed me to enter the LUKS password and log in. However, this caused Xorg and LightDM to fail, with Xorg repeatedly crashing.

It was clear this was an issue with the amdgpu module. I tried several kernel parameters without success:

  • amdgpu.modeset=1
  • amdgpu.dc=0
  • amdgpu.dpm=0
  • amdgpu.ppfeaturemask=0xffffb

Eventually, I found that amdgpu.dcdebugmask=0x10 resolved the problem.

qubesd Issues

After finally booting into the system, I couldn't attach any block devices, including my /boot partition, to dom0. This turned out to be a bug in qubesd, which I reported.

Backup Strategy

Backing up data in Qubes OS can be tricky due to its design. Here is the high-level strategy I've been planning:

  • Create as few templates as possible.
  • Only use Salt to configure templates. So I only need to back up Salt files in dom0.
  • Changes to dom0 are either managed by Salt, or the relevant files are included in the backup. Examples
    • Xfce settings.
    • Qube settings/features, /etc/qubes
    • /etc/default/grub
    • /etc/dracut.conf.d

My backup process is as follows:

  • I use a dedicated disposable VM with restic installed. The actual backup scripts and SSH keys remain in dom0.
  • Data from dom0 and other qubes is archived using tar. Use --transform to prepend the VM name to the path.
  • Use qvm-run --pass-io --no-gui to pass the scripts, SSH keys an data to the disposable VM, which then runs the scripts to execute the backup.

There are a few things to consider:

  • Using --pass-io could have security implications. It might be possible to mitigate this by limiting the number of bytes passed and saving output to a file.
  • The tar archives are currently extracted in the disposable VM. If the data isn't trusted, this could be a security risk. In such cases, the extraction step could be skipped.
  • Accessing the backed-up data requires starting a VM. An alternative could be to create an LVM snapshot and back that up directly.

Thought: Manging Qubes like Containers

There are many similarities between Qubes VMS and containers (docker, podman):

  • Template VMs are similar to building containers.
  • AppVMs are like running containers with persistent volumes.
  • Disposable VMs are like running containers without persistent volumes.

There are some gaps:

  • One container can be based on others, the shared part are often stored only once as layers. While cloning a template VM can use reflink for initial efficienty, changes will evetually cause the data to diverge. bootc might help bridge this gap.
  • Containers support bind mounts, which is very convenient for backup. While virtiofsd could work for Xen, but I guess there are security concerns. An alternative could be to centralize all data in one qube and share it with others over the network, but again, there could be security concerns.

Conclusion

It's been a fun and challenging journey exploring Qubes OS. While there were some hurdles with hardware and system upgrades, working through them has been a valuable learning experience. The security architecture is powerful, and I'm excited to continue finding new ways to make it work for my setup.

Labels:

Comments

Post a Comment

Popular posts from this blog

Determine Perspective Lines With Off-page Vanishing Point

In perspective drawing, a vanishing point represents a group of parallel lines, in other words, a direction. For any point on the paper, if we want a line towards the same direction (in the 3d space), we simply draw a line through it and the vanishing point. But sometimes the vanishing point is too far away, such that it is outside the paper/canvas. In this example, we have a point P and two perspective lines L1 and L2. The vanishing point VP is naturally the intersection of L1 and L2. The task is to draw a line through P and VP, without having VP on the paper. I am aware of a few traditional solutions: 1. Use extra pieces of paper such that we can extend L1 and L2 until we see VP. 2. Draw everything in a smaller scale, such that we can see both P and VP on the paper. Draw the line and scale everything back. 3. Draw a perspective grid using the Brewer Method. #1 and #2 might be quite practical. #3 may not guarantee a solution, unless we can measure distances/p...
3 comments
Read More »

Exploring Immutable Distros and Declarative Management

My current server setup, based on Debian Stable and Docker, has served me reliably for years. It's stable, familiar, and gets the job done. However, an intriguing article I revisited recently about Fedora CoreOS, rpm-ostree, and OSTree native containers sparked my curiosity and sent me down a rabbit hole exploring alternative approaches to system management. Could there be a better way? Core Goals & Requirements Before diving into new technologies, I wanted to define what "better" means for my use case: The base operating system must update automatically and reliably. Hosted services (applications) should be updatable either automatically or manually, depending on the service. Configuration and data files need to be easy to modify, and crucially, automatically tracked and backed up. Current Setup: Debian Stable + Docker My current infrastructure consists of several servers, all running Debian Stable. System Updates are andled automatically via unattended-upgrades. Se...
Post a Comment
Read More »

Qubes OS: First Impressions

A few days ago, while browsing security topics online, Qubes OS surfaced—whether via YouTube recommendations or search results, I can't recall precisely. Intrigued by its unique approach to security through compartmentalization, I delved into the documentation and watched some demos. My interest was piqued enough that I felt compelled to install it and give it a try firsthand. My overall first impression of Qubes OS is highly positive. Had I discovered it earlier, I might have reconsidered starting my hardware password manager project. Conceptually, Qubes OS is not much different from running a bunch of virtual machines simultaneously. However, its brilliance lies in the seamless desktop integration and the well-designed template system, making it far more user-friendly than a manual VM setup. I was particularly impressed by the concept of disposable VMs for temporary tasks and the clear separation of critical functions like networking (sys-net) and USB handling (sys-usb) into the...
Post a Comment
Read More »