Skip to main content

[Updated] Failed Attempt: NixOS in LXC

[Updates: 2024-10-21]

I found a working solution:

- Disable sandboxing in configuration.nix

- Build a tarball image using `nixos-generator -f lxc`

- Create an LXC container with `lxc-create -t none`

- Modify the config of the LXC container (e.g. specify rootfs path, set unconfined AppArmor)

- Create the rootfs directory and remove all POSIX ACL (setfacl --remove-all)

- Extract the tarball into rootfs/



I planned to try NixOS in LXC.

I have found a few successfull stories: 1, 2, 3. However they are all using Proxmox LXC, and/or the image file is for LXD.


First, I tried to download the official image via lxc-create. The image can boot, but I have trouble running `nix-channel --update`, which complains about sandboxing. I think it's related to unprivileged LXC containers. Further, as part of the nice feature of NixOS, I cannot easily disable sandbox from there.

Second, I tried to build a NixOS image from scratch, using nixos-generators. This is mentioned in the 3rd link above. This tools supports lxc format, however, lxc-create -t local does not like the metadata generated by this tool.


At this point I lost interests and decided the stop. The original idea is I might try NixOS longer if it is easier to maintain than Arch, but it doesn't seems so. If I have to make something work. Maybe I can use the generate image from option 2 to override the rootfs in option 1. Perhaps it'll work.




Labels:

Comments

Post a Comment

Popular posts from this blog

A Rocky Migration: Moving from docker-compose to Podman and gVisor

I've been running a few containers for several years. They were all running under rootless Docker with a single user. Initially, I planned to  migrate the containers to VMs , but I couldn't get a stable workflow after about two months of effort. Later,  gVisor caught my attention , and I decided to migrate to Podman with gVisor instead. The new plan is to run each container with  --userns=auto  and use Quadlet for systemd integration. This approach provides better isolation and makes writing firewall rules easier. I'm now close to migrating all my containers. Here are a couple of rough edges I'd like to share. Network Layout I compared  various networking options  and spent a few hours trying the one-interface-per-group approach before giving up. I settled on a single macvlan network and decided to use static IP addresses for my containers. To prevent a randomly assigned IP address from conflicting with a predefined one, I allocated a large IP range for my ...
Post a Comment
Read More »

Exploring Immutable Distros and Declarative Management

My current server setup, based on Debian Stable and Docker, has served me reliably for years. It's stable, familiar, and gets the job done. However, an intriguing article I revisited recently about Fedora CoreOS, rpm-ostree, and OSTree native containers sparked my curiosity and sent me down a rabbit hole exploring alternative approaches to system management. Could there be a better way? Core Goals & Requirements Before diving into new technologies, I wanted to define what "better" means for my use case: The base operating system must update automatically and reliably. Hosted services (applications) should be updatable either automatically or manually, depending on the service. Configuration and data files need to be easy to modify, and crucially, automatically tracked and backed up. Current Setup: Debian Stable + Docker My current infrastructure consists of several servers, all running Debian Stable. System Updates are andled automatically via unattended-upgrades. Se...
Post a Comment
Read More »

mkosi: First Impressions

I stumbled upon the Gentoo wiki page for systemd-nspawn , which in turn led me to nspawn.org , mkosi , and later systemd-sysupdate . mkosi quickly caught my eye because it's almost exactly what I wanted to build myself, as mentioned in a previous post . So, I decided to spend my "sysadmin fun quota" on it. Overview mkosi is similar to docker build or podman build , but it's designed for creating full OS images. It focuses on development and testing. For example, much like nix-shell , mkosi can quickly launch a sandboxed shell with a specific distribution and selected packages installed. The systemd project itself uses mkosi for testing across different distros. The re-introduction article  is a great read. Speed Note that this is by no means a rigid benchmark. My setup is an SSD with LUKS and an ext4 filesystem (without reflink support). Building Container Images mkosi is pretty fast. A simple mkosi command creates a fresh Debian image. I used the --incrementa...
Post a Comment
Read More »