A Journey into Podman: Notes on My First Adventure

For the last few days, I've been experimenting with Podman. My goal was to get a feel for the setup, create a minimal yet scalable environment for a few containers, and identify potential problems early on.

Here are my notes from this experience.

Quadlet

Quadlet allows you to define containers, networks and more using a syntax similar to systemd. This includes helpful features like drop-in overrides and templates.

The framework is tightly integrated with systemd, and Quadlet actually generates real systemd units. This means I can directly write systemd options in my Quadlet files.

One of the biggest benefits I've found is how easy Quadlet makes it to set up socket activation. This allows me to place some containers in an internal network or even without a network at all.

Hardening Defaults

Let's say I have a group of Systemd and Quadlet units, all named in the format of xyz-*. My goal is to define some secure, hardened default values for these units that can still be overridden by individual units.

For example, I want to change the default for ProtectSystem= from false to true.

Simply creating a xyz-.service.d/00-override.conf doesn't work, because an individual unit cannot override this setting. While I could create a xyz-foo.service.d/10-override.conf for each specific service, this would split the definition of xyz-foo into two separate files, which isn't ideal.

To solve this, I created a script that moves the main xyz-foo.service file to xyz-foo.service.d/10-override.conf and creates a nearly empty xyz-foo.service as a placeholder. This facilitates the process of setting and overriding defaults.

gVisor

Setting up gVisor's runsc was straightforward, and I haven't encountered any compatibility issues so far.

Unfortunately, the version of gVisor in the Debian repositories is quite old, and Debian is unable to provide prompt security updates for it. This meant I had to add the official gVisor apt repository. It's not the ideal solution, but it works.

Credentials

Systemd-Credentials is a very handy tool for managing sensitive information and can be used directly in Quadlet files.

However, I ran into an issue when using it with containers that have --userns=auto. Because Podman still runs as root, the credentials are only readable by the root user and not by the containers.

Podman offers its own solution for this called podman-secret. This feature allows you to either have Podman store the secrets or use drivers to connect to your own storage solution.

I prefer to keep all my secrets in a dedicated directory. To accommodate this, I wrote a simple script that registers a file as a Podman secret:

podman secret create \
  --driver=shell \
  --driver-opts=list="echo $1" \
  --driver-opts=lookup="cat $2" \
  --driver-opts=store=/bin/true \
  --driver-opts=delete=/bin/true \
  --replace=true \
  "$1" - <<<SECRET

For any container that needs access to secrets, I simply call this script in ExecStartPre=.

Networking

My plan is to run a few groups of containers with the following requirements:

• Containers within the same group can communicate with each other, but containers from different groups cannot.
• I want to avoid manually managing IP addresses for each group or container.
• It should be easy to write firewall rules to intercept container traffic.

I explored a few options to achieve this:

1. Plain Bridges: The simplest approach is to create a default bridge for each container group, which is the default behavior in Podman. Everything works out of the box, and I don't need to explicitly specify IP addresses. However, writing firewall rules is tricky because the IP addresses and bridge names are not predetermined. To make this work, I would need to carefully define an IP range and/or bridge name for each bridge.

2. Single Unmanaged Bridge: Another option is to create a bridge using systemd-networkd and then, for each container group, create a Podman bridge network using the existing bridge with VLANs and/or isolation. This didn't work well. Since the bridge is unmanaged, I would need to manually set up DHCP, DNS, and the firewall. My attempts to simply set up a DHCP server on the bridge were unsuccessful. See also 1, 2

3. VRF: I tried creating a VRF with systemd-networkd and then creating a Podman bridge network for each container group, specifying the VRF. This was a promising and straightforward solution that met most of my requirements, except for one critical issue: DNS server doesn't work. While I was able to force aardvark-dns to use the VRF by using a wrapper script (exec ip vrf exec MY-VRF aardvark-dns "$@"), this solution felt too fragile for long-term use.

4. Bridge + veth + macvlan: This more complex setup involves creating a bridge as in the second option, then a veth pair for each container group, and finally using that to create a Podman macvlan network. Theoretically, this should work, but it offers no significant advantages over the VRF option and is more complicated to configure. It's also worth noting that the DNS server is not supported in macvlan mode.

In the end, I decided to go with option 1. It's simple, and it works. I might switch to option 3 when the built-in DNS server supports VRF, or when I don't need a DNS server.

Final Thoughts

So far, the combination of Podman, Quadlet, and gVisor has been a positive experience. Not everything has worked perfectly, but I'm quite happy with the setup. If things continue to go well, I might be able to migrate my docker-compose setup in the near future.