WangLu's Notes

The central question is: how do you manage secrets like SSH keys, API keys, and passwords for disposable VMs ? 🤷‍♂️ Let's establish some ground rules for this scenario. Suppose I want to pass an API key to the VM  chimera , which is run by the  chimera-runner user on the host. My security requirements are: On the host, only root and  chimera-runner  should have access to the secrets. In VM  chimera , only root and relevant service users should have access to the secrets. No one from other VMs, including their root users, should have access to VM  chimera 's secrets. The guest VMs themselves are not trusted . The bootc documentation on this topic is very informative. On a high level, there are a few ways to achieve this. 1. OEM Strings / Firmware QEMU can pass data to a VM via SMBIOS OEM strings ( -smbios ) or firmware configuration ( -fw_cfg ). Notably, both methods are supported by systemd-creds using special keys. This approach is practical for sma...

[Update] I managed to work out the AppArmor profile, and decided to go with guestmount for now. I am setting up a maintenance pipeline for my virtual machines . The pipeline has two main routines: The BACKUP routine: Every day, this routine shuts down each VM, backs up the data in /var, updates the VM's disk image if a new version is available, and then restarts it. The BUILD routine: Every week, this routine uses a special builder VM to create new disk images for all VMs. There is a scheduling conflict with the builder VM: the BACKUP routine needs to shut it down, while the BUILD routine needs it running. To resolve this, I merged both into a single set of systemd services that runs daily. The BUILD routine starts automatically when the builder VM starts, at the end of the BACKUP routine. The builder VM's systemd unit has an ExecCondition= property, which is skipped 6 days a week. Surprisingly, the most difficult part of this pipeline was not the scheduling, but the backup pro...

Just learned about GNU Stow , which is a tool for managing symlink farm. Basically the idea is to store all files in one place, then create symlink all around the system pointing to your files. There are various use cases, like dot files and installing/uninstalling packages. But I mostly use it for tracking system config files, similar to how NixOS works. In fact I wrote my own scripts with "cp -rs", but GNU Stow works much better.

Today, various services (native, LXC, Docker) are running on my server. I'm mostly happy with the setup, but I decided to revisit my server's defenses under the assumption that a remote attacker or malicious code could compromise my services. A service might break out of its container or even gain root privilege. VMs are a better security boundary than containers; they can limit the damage if an attacker gains root privilege. I cannot afford to run a dedicated VM for each service, so I will need to carefully group the services and run a dedicated VM for each group. Each group should be carefully designed based on the data accessed and the features/capabilities required. For example, some VMs may have access to my photos, while others may not have network access. The Goal There are two particular issues I want to address: First, I want VM images to be easily reproducible, which makes backup and restore trivial. NixOS and GNU Guix System are great examples, where you only need to...

Just learned about a few interesting and useful stuff, when playing with bootc: systemd Components systemd-tmpfiles  and systemd-sysusers  allows managing files and users in a declarative way. Originally I learned about this for building bootc images, but later I realized that they are also very useful on Debian. I learned systemd-credential  as a way of passing ssh authorized keys to a QEMU VM, but after reading more, I realized it can be used in other interesting ways. My favorite one is with LoadCredential=, I can run a script with DynamicUser=yes and the script can access some root-only secrets. I finally decided to migrate from cron to systemd-timer. systemd-timer is more interesting and handy than expected, and the migration process is less painful than expected. SELinux Actually I heared about SELinux many years ago. Over the time I just know SELinux as "something about security, similar but more complicated to AppArmor". Recently I got to learn more about it: - ht...

I decided to spend some time playing with bootc. Mostly I'm inspired by the following articles:  CoreOS + native container Hand-on demo (the last video), build bootc and auto update from registry bootc desktop bootc for homelab Day 1 To install bootc in a VM I need an image. bootc-image-builder requires root and I don't want to run this on the host. So I chose CoreOS as the inital system and installed it to QEMU. I thought it is a great idea to share a folder from host to guest as podman container storage. However, it was not as smooth as I had expected: virtiofsd on Debian is too old, so I set up NFS. rootless podman doesn't work well with NFS . rootfull podman complains upstream fs of overlayfs missing features, the performance was terrible. I gave up. I guess I'll just use the CoreOS disk, whose size is 10G, not enough. Day 2 I didn't find a way of resizing a qcow2 image online. On the other hand I figured maybe I don't need build a disk image after all. Cor...

When I have important data on a device, I back it up to my server using dedicated user accounts. The other day, I checked /etc/passwd on my server and found entries like this: some-backup-user1:x:1003:1004:... some-backup-user2:x:1004:1007:... A few inconsistencies immediately bothered me: UID/GID Mismatches: Many users have UIDs that don't match their primary GIDs. While this technically works and might seem like just an aesthetic concern, I realized that UIDs and GIDs are crucial metadata. I need to preserve them accurately for future system migrations to maintain correct file ownership.ID Ambiguity: ID Ambiguity: The same number (e.g., 1004) could represent a User ID for one account and a Group ID for a completely different group. This overlap is a recipe for mistakes during administration tasks if I'm not paying close attention. Lack of Structure: User and group accounts created for very different purposes – regular logins, backup processes, container management, specifi...

My current server setup, based on Debian Stable and Docker, has served me reliably for years. It's stable, familiar, and gets the job done. However, an intriguing article I revisited recently about Fedora CoreOS, rpm-ostree, and OSTree native containers sparked my curiosity and sent me down a rabbit hole exploring alternative approaches to system management. Could there be a better way? Core Goals & Requirements Before diving into new technologies, I wanted to define what "better" means for my use case: The base operating system must update automatically and reliably. Hosted services (applications) should be updatable either automatically or manually, depending on the service. Configuration and data files need to be easy to modify, and crucially, automatically tracked and backed up. Current Setup: Debian Stable + Docker My current infrastructure consists of several servers, all running Debian Stable. System Updates are andled automatically via unattended-upgrades. Se...

A few days ago, while browsing security topics online, Qubes OS surfaced—whether via YouTube recommendations or search results, I can't recall precisely. Intrigued by its unique approach to security through compartmentalization, I delved into the documentation and watched some demos. My interest was piqued enough that I felt compelled to install it and give it a try firsthand. My overall first impression of Qubes OS is highly positive. Had I discovered it earlier, I might have reconsidered starting my hardware password manager project. Conceptually, Qubes OS is not much different from running a bunch of virtual machines simultaneously. However, its brilliance lies in the seamless desktop integration and the well-designed template system, making it far more user-friendly than a manual VM setup. I was particularly impressed by the concept of disposable VMs for temporary tasks and the clear separation of critical functions like networking (sys-net) and USB handling (sys-usb) into the...

For the past month, I've been prototyping my own hardware password manager using the Cardputer , a compact device that's surprisingly perfect for the task.  I learned about the Cardputer while searching for the ideal hardware to build a password manager. It's essentially a microcontroller (the ESP32-S3) packed with a screen and a keyboard. What really sold me on it were these features: Direct Interaction: I can interact directly with the device – typing my master password, searching for credentials, and confirming password entry, all on the Cardputer itself. USB Keyboard Emulation: The device can seamlessly emulate a USB keyboard, allowing for both manual and automatic password entry into other devices. Hardware-Accelerated Crypto: The ESP32-S3 boasts hardware acceleration for AES and SHA operations, crucial for secure password management. Secure Element Integration:  The Cardputer supports an optional ATECC608B secure element (available as an official accessory ), provid...

Flash encryption and secure boot are useful security features for ESP32S3 chip. While not perfect, they definitely make it harder to extract the secrets in the chip. However, it is tricky to enable both features at the same time. The topic is actually discussed in the official documentation: ESP32S3 Security Features Security Features Enablement Workflows Especially, the second one mentioned it is recommended to enable flash encryption before secure boot. But I still find the documentation confusing. In the end I was able to successfully enable both, here's my findings. My Understanding After my adventure, here's what I think could have worked. WARNING, this is untested. Follow  Security Features Enablement Workflows : Burn all the keys, as long as their purpose eFuses and read/write protections Burn other security eFuses, but DO NOT burn ENABLE_SECURITY_DOWNLOAD in the middle, which is mentined at the end of the instruction for both flash encryption and secure boot. Burn...

[Updates 2025-01-20] The original blog post assumes that all passwords are stored in one password manager, and the password manager either unlocks everything or nothing (e.g. Keepass). After discussing with friends, I realized that if I use something like pass , and I use a hardware GPG token, I can actually store and sync all encrypted passwords to all devices, because I will only decrypt the passwords on demand, and the computer will not see the GPG private key. The compromise is that the computer will see the list of all password entries (e.g. accounts), as well as a few other issues . I've been using Keepass for many years. I don't use online password services because I cannot fully trust them. Besides, I may not always have Internet connection, which is why I also don't use a self-hosted service. Everything has been working fine, until I turn my paranoid knob to the max. Here's the thought experiment . The Imaginary Scenario Let's say I have 100 PCs for differe...

Windows 10 will soon reach its end of life, and my 1-gen Surface Pro is not supported by Windows 11. I (finally) decided to install Linux to it. Fortunately, it's a not-so-easy nice adventure: The device has only one USB port, so I have to bring back my 10+-year old USB hub. My live USB drive cannot boot directly, I have to disable Secure Boot first, by holding Volume Up during boot. I think years ago I learned that booting on USB might not work through a USB hub, but fortunatelly it worked well with my setup. This is done by holding Volume Down during boot. Wifi adapter was detected in the live Linux environment, but not functional. And I don't have a USB-Ethernet adapter. Luckily, nowadays we have USB-tethering from Android phones, which works out-of-the-box. Originally I planned to following this guide to set up root on ZFS, however, the system froze when building the ZFS kernel module. Then I decided to just use EXT4, yet I still learned a lot from the guide about disk par...