Skip to main content

Disposable VMs for Home Lab Security and Reproducibility

Today, various services (native, LXC, Docker) are running on my server. I'm mostly happy with the setup, but I decided to revisit my server's defenses under the assumption that a remote attacker or malicious code could compromise my services. A service might break out of its container or even gain root privilege.

VMs are a better security boundary than containers; they can limit the damage if an attacker gains root privilege. I cannot afford to run a dedicated VM for each service, so I will need to carefully group the services and run a dedicated VM for each group. Each group should be carefully designed based on the data accessed and the features/capabilities required. For example, some VMs may have access to my photos, while others may not have network access.


The Goal

There are two particular issues I want to address:

First, I want VM images to be easily reproducible, which makes backup and restore trivial. NixOS and GNU Guix System are great examples, where you only need to back up the configuration file. However, I don't really like them because of their domain-specific language/design.

Second, I want to seal the system as much as possible. Even a compromised root user inside a VM should not be able to permanently infect the VM. Many so-called "immutable" Linux distributions are not truly 100% immutable. Often, they just mean a read-only /usr. Some can be easily broken via `mount -o remount,rw`, and most of them allow self-upgrade, meaning a malicious root user can still inject code via "upgrade and reboot."

The Approach

I use bootc containers. This allows me to build the whole system with standard scripts, and it offers the standard "immutability."

Furthermore, I run QEMU with `--no-reboot --snapshot`, which means the system cannot update itself even with root privilege.

Lastly, I'll regularly build new images and restart the VM to pick up the latest security fixes.


This approach is essentially managing VMs like containers. It's not a new idea; frood and gokrazy are good examples of this principle.

On a side note, I also plan to learn more about KubeVirt and Nix VMs. Especially, I like the idea (from NixOS) that the guest can directly use the store from the host.

Notes about QEMU

Permanent machine-local data is stored in /var, which is put into a separate disk image.

Secrets are sent to QEMU via systemd credentials.

I tried virtiofsd, but didn't like it. I ended up with Samba anyway. Maybe I'll revisit virtiofsd later.

To shut down the VM (e.g., via systemd), I created a special admin user with special privilege defined in the sudoers file, so that I can run `ssh admin@vm sudo poweroff`. The SSH key pair is regenerated before each VM boot. Related: In a systemd unit, ExecStop= does not have access to LoadCredential.

I use `-chardev socket,logfile=...` and `-serial` so that the systemd logs are not filled with console output, and I can view or attach to the serial console later.

I plan to learn more about virtio-balloon and pmem later.

Conclusion

I find it very beneficial to deploy VMs. It allows me to shrink and harden the host OS (e.g., disable unprivileged user namespaces), and it allows me to design fine-grained access control.

Next, I'll start investigating how to organize the containers inside VMs. 

Labels:

Comments

Post a Comment

Popular posts from this blog

Determine Perspective Lines With Off-page Vanishing Point

In perspective drawing, a vanishing point represents a group of parallel lines, in other words, a direction. For any point on the paper, if we want a line towards the same direction (in the 3d space), we simply draw a line through it and the vanishing point. But sometimes the vanishing point is too far away, such that it is outside the paper/canvas. In this example, we have a point P and two perspective lines L1 and L2. The vanishing point VP is naturally the intersection of L1 and L2. The task is to draw a line through P and VP, without having VP on the paper. I am aware of a few traditional solutions: 1. Use extra pieces of paper such that we can extend L1 and L2 until we see VP. 2. Draw everything in a smaller scale, such that we can see both P and VP on the paper. Draw the line and scale everything back. 3. Draw a perspective grid using the Brewer Method. #1 and #2 might be quite practical. #3 may not guarantee a solution, unless we can measure distances/p...
3 comments
Read More »

Qubes OS: First Impressions

A few days ago, while browsing security topics online, Qubes OS surfaced—whether via YouTube recommendations or search results, I can't recall precisely. Intrigued by its unique approach to security through compartmentalization, I delved into the documentation and watched some demos. My interest was piqued enough that I felt compelled to install it and give it a try firsthand. My overall first impression of Qubes OS is highly positive. Had I discovered it earlier, I might have reconsidered starting my hardware password manager project. Conceptually, Qubes OS is not much different from running a bunch of virtual machines simultaneously. However, its brilliance lies in the seamless desktop integration and the well-designed template system, making it far more user-friendly than a manual VM setup. I was particularly impressed by the concept of disposable VMs for temporary tasks and the clear separation of critical functions like networking (sys-net) and USB handling (sys-usb) into the...
Post a Comment
Read More »

ESP32S3: Flash Encryption and Secure Boot

Flash encryption and secure boot are useful security features for ESP32S3 chip. While not perfect, they definitely make it harder to extract the secrets in the chip. However, it is tricky to enable both features at the same time. The topic is actually discussed in the official documentation: ESP32S3 Security Features Security Features Enablement Workflows Especially, the second one mentioned it is recommended to enable flash encryption before secure boot. But I still find the documentation confusing. In the end I was able to successfully enable both, here's my findings. My Understanding After my adventure, here's what I think could have worked. WARNING, this is untested. Follow  Security Features Enablement Workflows : Burn all the keys, as long as their purpose eFuses and read/write protections Burn other security eFuses, but DO NOT burn ENABLE_SECURITY_DOWNLOAD in the middle, which is mentined at the end of the instruction for both flash encryption and secure boot. Burn...
Post a Comment
Read More »