A Practical Guide to Passing Secrets to VMs

The central question is: how do you manage secrets like SSH keys, API keys, and passwords for disposable VMs? 🤷‍♂️

Let's establish some ground rules for this scenario. Suppose I want to pass an API key to the VM chimera, which is run by the chimera-runner user on the host. My security requirements are:

• On the host, only root and chimera-runner should have access to the secrets.

• In VM chimera, only root and relevant service users should have access to the secrets.

• No one from other VMs, including their root users, should have access to VM chimera's secrets.

• The guest VMs themselves are not trusted.

The bootc documentation on this topic is very informative.

On a high level, there are a few ways to achieve this.

1. OEM Strings / Firmware

QEMU can pass data to a VM via SMBIOS OEM strings (-smbios) or firmware configuration (-fw_cfg). Notably, both methods are supported by systemd-creds using special keys.

This approach is practical for small pieces of data, like an individual password or an encryption key. It's not a new idea and was discussed years ago.

However, there are a few caveats:

• Size Limits: The QEMU manpage states that the total size of all SMBIOS tables is limited to 65535 bytes. While not explicitly defined, fw_cfg is also intended for small amounts of data.

• Key Length: The maximum length of a fw_cfg key name is 55 characters. If you use it with systemd-creds, a special prefix is required, making the available space for your key name even shorter.

• Security Risk: If you pass data as a string (e.g. -fw_cfg string=secrets), the secret becomes part of the QEMU process's command line, which is visible to all users on the host!

• Bugs: You can provide a file to SMBIOS using -smbios path=filename, which avoids exposing the secret on the command line. However, this feature is affected by a bug that is still present in Debian Bookworm.

• Accessibility: Data passed via firmware appears in the guest's /sys/firmware directory, which cannot be mounted as a typical block device.

For these reasons, I generally use -fw_cfg file=filepath for passing small secrets and leverage systemd-creds within the guest whenever possible.

I might switch to -smbios path=filename later, when the bug is fixed.

2. Network Share

This is probably the most common way of sharing files between a host and its guests. The host sets up a file-sharing service, and the guest connects to it to access the files.

• Pros: Changes on the host are reflected in the guest immediately, although this might not be important for static secrets.

• Cons: The server on the host needs to authenticate clients (the VMs) to ensure one VM cannot access another's secrets. The guest VM also needs to set proper file permissions internally.

Common options include:

• QEMU's built-in SMB server: Easy to set up, but the guest share is accessible to everyone in the guest. A non-root user can access the content using userspace tools.

• SMB Server: For each VM, you must create a dedicated user/password pair. This password must then be passed securely to the VM (using another method).

• NFS Server: NFS authenticates clients by IP address and trusts the client's root user. This is risky because a compromised VM could spoof its IP address. An extra authentication layer, like WireGuard, might be necessary.

• SSHFS: Each VM needs a dedicated SSH key pair stored securely. The host can use a standard SSH server configuration.

While these options are viable, I find them less than ideal. They require significant effort to set up correctly, and the servers add extra maintenance overhead. Furthermore, SSHFS appears to be no longer actively maintained.

3. Filesystem Passthrough

This method is similar to a network share but is optimized for VM environments. Instead of the network stack, it uses a more direct channel to expose a host filesystem to the guest, which is generally faster.

Common options are:

• 9pfs: Easy to set up, but my guest OS (centos-bootc:stream9) lacks the necessary kernel support, and I prefer not to compile custom kernels.

• virtiofsd: This is a popular and high-performance method, but it requires shared memory between the host and guest, plus an extra daemon running on the host.

Unfortunately, neither of these options worked for my specific setup.

4. Credential Fetcher

With this approach, the guest fetches credentials as needed, typically during boot. This can be implemented easily in the guest, for example, by using scp to copy secrets into a ramfs mount.

However, this requires setting up a server (e.g. SSH) on the host, which I wanted to avoid due to the added complexity and maintenance.

5. Embed in Container Image

It's possible to generate and embed secrets directly into the bootc container image during the build process.

An interesting variation is to encrypt the secrets, embed the encrypted data in the image, and pass the decryption key to the VM using another method (like an OEM string). You could use systemd-creds for this and even bind decryption to a virtual TPM simulated by QEMU. However, as noted in this discussion, this might not be the intended use case for systemd-creds.

While this works in theory, it doesn't offer significant benefits for my workflow and feels tedious to implement.

6. Disk Images

Any file can be used as a raw disk image for the VM, the guest just directly read the device for the data, without mounting a filesystem. Note that there may be issues if the file size is not aligned with the block size.

A more practical approach is to create a proper disk image containing the secrets and attach it to the VM. Any filesystem supported by the guest OS will work, but the ideal choice would have these characteristics:

• The disk image can be created without root privileges on the host.

• The filesystem is optimized for read-only data.

I found three practical options:

• EROFS: A modern, read-only filesystem that supports volume labels.

• SquashFS: mkfs.squashfs is very flexible; its "pseudo file" feature lets you create files directly from command output.

• ISO 9660 (CD/DVD images): Universally supported but less flexible.

I plan to use EROFS mainly because it supports volume labels. I cannot guarantee that the disk order will be consistent across all VMs. Therefore, the guest needs a reliable way to identify the secrets disk, and mounting by label is the easiest solution.

Conclusion

After evaluating the options, I settled on the following combination:

• OEM strings for small, individual secrets (like a decryption key).

• Read-only EROFS disk images for larger sets of secret files.

• QEMU's built-in SMB server for sharing encrypted data blobs, where the decryption key is passed separately via an OEM string.

Keep in mind that this solution is tailored to my specific use case, which has the following constraints:

• The guest VMs are not fully trusted.

• I want to minimize setting up and maintaining complex services on the host.

• Automating the setup via scripting before a VM starts is acceptable and even preferable.

• The solution must scale easily to many different VMs.