A Journey into Podman: Notes on My First Adventure

For the last few days, I've been experimenting with Podman. My goal was to get a feel for the setup, create a minimal yet scalable environment for a few containers, and identify potential problems early on.

Here are my notes from this experience.

[Updates 2025-09-21] Added more networking options and other information.

Quadlet

Quadlet allows you to define containers, networks and more using a syntax similar to systemd. This includes helpful features like drop-in overrides and templates.

The framework is tightly integrated with systemd, and Quadlet actually generates real systemd units. This means I can directly write systemd options in my Quadlet files.

One of the biggest benefits I've found is how easy Quadlet makes it to set up socket activation. This allows me to place some containers in an internal network or even without a network at all.

Hardening Defaults

Let's say I have a group of Systemd and Quadlet units, all named in the format of xyz-*. My goal is to define some secure, hardened default values for these units that can still be overridden by individual units.

For example, I want to change the default for ProtectSystem= from false to true.

Simply creating a xyz-.service.d/00-override.conf doesn't work, because an individual unit cannot override this setting. While I could create a xyz-foo.service.d/10-override.conf for each specific service, this would split the definition of xyz-foo into two separate files, which isn't ideal.

To solve this, I created a script that moves the main xyz-foo.service file to xyz-foo.service.d/10-override.conf and creates a nearly empty xyz-foo.service as a placeholder. This facilitates the process of setting and overriding defaults.

gVisor

Setting up gVisor's runsc was straightforward, and I haven't encountered any compatibility issues so far.

Unfortunately, the version of gVisor in the Debian repositories is quite old, and Debian is unable to provide prompt security updates for it. This meant I had to add the official gVisor apt repository. It's not the ideal solution, but it works.

Credentials

Systemd-Credentials is a very handy tool for managing sensitive information and can be used directly in Quadlet files.

However, I ran into an issue when using it with containers that have --userns=auto. Because Podman still runs as root, the credentials are only readable by the root user and not by the containers.

Podman offers its own solution for this called podman-secret. This feature allows you to either have Podman store the secrets or use drivers to connect to your own storage solution.

I prefer to keep all my secrets in a dedicated directory. To accommodate this, I wrote a simple script that registers a file as a Podman secret:

podman secret create \
  --driver=shell \
  --driver-opts=list="echo $1" \
  --driver-opts=lookup="cat $2" \
  --driver-opts=store=/bin/true \
  --driver-opts=delete=/bin/true \
  --replace=true \
  "$1" - <<<SECRET

For any container that needs access to secrets, I simply call this script in ExecStartPre=.

I found a few issues, and create an issue on GitHub. Fortunately workarounds are available.

Networking

My plan is to run a few groups of containers with the following requirements:

• Containers within the same group can communicate with each other, but containers from different groups cannot.
• I want to avoid manually managing IP addresses for each group or container.
• It should be easy to write firewall rules to intercept container traffic.

I explored a few options to achieve this:

1. Plain Bridges: The simplest approach is to create a default bridge for each container group, which is the default behavior in Podman. Everything works out of the box, and I don't need to explicitly specify IP addresses. However, writing firewall rules is tricky because the IP addresses and bridge names are not predetermined. To make this work, I would need to carefully define an IP range and/or bridge name for each bridge.

2. Single Bridge: Another option is to create a bridge using systemd-networkd and then, for each container group, create a Podman bridge network using the existing bridge with VLANs and/or isolation. Since the bridge is unmanaged, I will need to manually set up DHCP, DNS, and the firewall.

   • Simply setting up a DHCP server on the bridge (i.e. in bridge.netweork) doesn't work. I think it is because podman, or more precisely, netavark-dhcp-proxy, will use the interface (bridge in this case) as DHCP client rather than server.
   • It works if I add an external DHCP server to the bridge. E.g. create a veth pair, put one end under the bridge and add a DHCP server on the other end.

3. VRF: I tried creating a VRF with systemd-networkd and then creating a Podman bridge network for each container group, specifying the VRF. DHCP works with this setup, but DNS server doesn't. While I was able to force aardvark-dns to use the VRF by using a wrapper script (exec ip vrf exec MY-VRF aardvark-dns "$@"), this solution felt too fragile for long-term use. Also, I wasn't able to easily isolate containers within the same network.

4. veth + macvlan: This setup involves creating a veth pair for each container group, putting DHCP server on one end, and then using the other end to create a Podman macvlan network. This works, and it is easy to isolate containers within one podman network, by using the private mode and setting up necessary firewall rules. But note that the DNS server is not supported in macvlan mode.

My Networking Plan

All these options more-or-less work, with different trade-offs. I plan to use 1 and 4 for different use cases:

1. When a group of containers need to find each other by name, I put them into a bridge network (option 1) with Internal=true. Both DCHP and DNS work, and I don't need to write much firewall rules.
2. When a container needs to access Internet, I create a private macvlan network (option 4) and put the container into it. Since I can easily isolate containers within the same network, it is possible to put containers from different groups into the same bridge network. Then I can just use the bridge network to group containers by policies, e.g. some containers can access everything, but some are only allows to visit specific URLs. I just need to write firewall rules for each bridge network.

Final Thoughts

So far, the combination of Podman, Quadlet, and gVisor has been a positive experience. Not everything has worked perfectly, but I'm quite happy with the setup. If things continue to go well, I might be able to migrate my docker-compose setup in the near future.