Exploring Immutable Distros and Declarative Management

My current server setup, based on Debian Stable and Docker, has served me reliably for years. It's stable, familiar, and gets the job done. However, an intriguing article I revisited recently about Fedora CoreOS, rpm-ostree, and OSTree native containers sparked my curiosity and sent me down a rabbit hole exploring alternative approaches to system management. Could there be a better way?

Core Goals & Requirements

Before diving into new technologies, I wanted to define what "better" means for my use case:

• The base operating system must update automatically and reliably.
• Hosted services (applications) should be updatable either automatically or manually, depending on the service.
• Configuration and data files need to be easy to modify, and crucially, automatically tracked and backed up.

Current Setup: Debian Stable + Docker

My current infrastructure consists of several servers, all running Debian Stable.

• System Updates are andled automatically via unattended-upgrades.
• Services consist of a mix of native Debian packages and applications running in rootless Docker containers. Docker images are updated either periodically via scripts or manually when needed.
• Config and data files are backed up automatically using rsync, but tracking which files need backing up is manual. This is my main pain point. Every time I add or modify a service or configuration file, I have to remember to update my backup scripts.

While this setup works, it's not perfect:

• The manual tracking of configuration files is error-prone and tedious. I like how systems like NixOS manage the entire system declaratively from configuration files, automatically tracking changes. However, I have reservations about NixOS's learning curve and ecosystem, and I generally prefer sticking to more "traditional" distributions like Debian if possible.
• Some services are Internet-facing. While rootless Docker improves security over rootful Docker, it's not a full security boundary. Moving these services into dedicated VMs could offer better isolation.

Fedora CoreOS + OSTree Native Container

Fedora CoreOS is an automatically updating, minimal, monolithic, container-focused operating system. Key concepts here include:

• The core OS is largely read-only, making updates safer and more predictable (atomic rollbacks).
• Configuration is applied on the first boot using Butane/Ignition. This is great for initial setup but less ideal for ongoing configuration changes.
• rpm-ostree allows layering additional RPM packages onto the immutable base image. While OSTree tracks file changes, rpm-ostree itself doesn't inherently provide a declarative way to manage the list of installed packages or track arbitrary configuration file modifications in a user-friendly, declarative manifest like NixOS does. It knows files changed, but not necessarily why in a structured way (e.g., "/usr/bin/vim was added" vs "package vim was installed").
• OSTree Native Containers: The article that inspired me highlighted using OSTree's ability to pull container images as system updates. The workflow looks like this:
• Define a system image using a Containerfile/Dockerfile, starting from a CoreOS base.
• Build this definition into an OCI container image.
• Install a standard CoreOS, using a minimal Butane config that tells rpm-ostree to "rebase" onto your custom container image.
• To update the system or add packages, modify the Containerfile, rebuild the image, push it to a registry, and the CoreOS systems will eventually pull and apply it as their next OS update.

This approach essentially creates a custom, version-controlled OS image. Variations include building the image via CI/CD (like quay.io), building locally, or even using coreos-assembler with a lower-level "treefile" manifest.

Bootable Containers

Emerging from the CoreOS world is Fedora Bootc. This seems specifically designed for use cases requiring more customization than standard CoreOS allows. Instead of Ignition and rpm-ostree, bootc directly manages bootable container images derived from a Containerfile. You essentially build your entire OS, including customizations, as a container image, and the system boots directly into it and updates by pulling new image versions. This feels conceptually cleaner and more aligned with the goal of a declaratively built system image.

Cloud-init

cloud-init is the de facto standard for bootstrapping cloud instances across various Linux distributions. Like Butane/Ignition, it runs on first boot, but critically, it can be re-run on subsequent boots. This makes testing configuration changes much easier, as you don't necessarily need to re-image the entire machine for every small tweak. It's widely supported but focuses more on initial setup and less on managing the entire OS lifecycle declaratively like the OSTree/Bootc approaches.

cloud-init is a popular tool to bootstrap linux machines for cloud. So it is similar to butane for CoreOS. However, cloud-init can be re-run, which means testing a simple change does not require re-imaging the machine.

Guix

Guix System is the GNU project's take on a declarative operating system, similar in principle to NixOS. It uses Guile/Scheme for its configuration language and boasts a clean command-line interface. While appealing from a purity perspective, its smaller community and adoption compared to NixOS make it a less pragmatic choice for me currently. Like NixOS, it can build VM images and mange VMs.

Conclusion and Thoughts

• Conceptually, Fedora Bootc is the most compelling alternative I found. It directly addresses the desire to build a customized, yet manageable and updatable, system image using familiar container tooling. The main drawback? It's very new, lacks widespread adoption, and crucially, there's no "Debian Bootc" yet. If a stable, Debian-based bootc implementation existed, I'd likely jump on it.
• Cloud-init is mature, widely adopted, and works across many distributions (including Debian). It could improve my bootstrapping process, but it doesn't solve the core desire for managing the entire system state declaratively post-install.
• CoreOS/OSTree: While powerful, the standard CoreOS workflow with rpm-ostree layering feels slightly less integrated than bootc for building a heavily customized system declaratively. The native container approach is interesting but adds complexity.
• NixOS/Guix/Ansible: These are powerful, but represent a significant shift in tooling and philosophy. They feel like a larger commitment than I'm ready for, especially given my preference for sticking closer to traditional distribution paradigms if possible.

For the time being, I'll likely stick with my current Debian + Docker setup. On the other hand, I might as well start a VM and try some options for better understanding.