Skip to main content

Backing Up VM Disk Images

[Update] I managed to work out the AppArmor profile, and decided to go with guestmount for now.

I am setting up a maintenance pipeline for my virtual machines.

The pipeline has two main routines:

  1. The BACKUP routine: Every day, this routine shuts down each VM, backs up the data in /var, updates the VM's disk image if a new version is available, and then restarts it.
  2. The BUILD routine: Every week, this routine uses a special builder VM to create new disk images for all VMs.
There is a scheduling conflict with the builder VM: the BACKUP routine needs to shut it down, while the BUILD routine needs it running. To resolve this, I merged both into a single set of systemd services that runs daily. The BUILD routine starts automatically when the builder VM starts, at the end of the BACKUP routine. The builder VM's systemd unit has an ExecCondition= property, which is skipped 6 days a week.

Surprisingly, the most difficult part of this pipeline was not the scheduling, but the backup process itself.

There are two general approaches to backing up a VM: backing up the entire disk image or backing up the files directly from the filesystem.

Backing Up Disk Images

Backing up a disk image is straightforward because disk images are just regular files. However, this method is often inefficient:
  • Deduplication may not work well if the disk image is compressed.
  • Incremental backups are not natively supported.
  • The entire disk image must be backed up, including unused and deleted data.
  • You cannot easily choose which specific files to include or exclude from the backup.
There are some ways to improve this approach:
  • For deduplication: I can decompress the disk image and pipe the data stream directly to the backup software without saving the decompressed file.
  • For incremental backups: I can create snapshots and back up only the differences. I would also need to regularly merge the snapshots.
    • QEMU supports incremental backup for a running VM. Relavent article: 1, 2.
  • To reduce backup size: I can defragment, shrink, wipe, and sparsify the disk image (for example, with virt-sparsify) before backing it up.
  • To exclude specific files: I can put the files or directories that I don't want to back up on a separate disk image.

Backing Up Filesystems

One can back up files from inside the VM. It is also possible to mount the disk image from the host system when the VM is shut down.

Raw disk images can be mounted directly using loop devices.

Qcow2 images have several options:
  • `qemu-nbd` can expose an image as a block device (e.g., /dev/nbd*), which can then be mounted. This requires the nbd kernel module and root access. The qemu-nbd man page warns that this may not be suitable for untrusted guests. To back up multiple VMs, I would also need a way to find an available /dev/nbd* device.
    • The block device can be exported without root, there are tools like `qemu-nbd`, `nbdfuse` and `qemu-storage-daemon`. This article is worth reading.
    • `qemu-nbd` also supports exposing an internal snapshot of a qcow2 image.
  • `guestmount` can mount a disk image without needing root. It uses QEMU and FUSE. While it works, it can be slow, and creating a secure AppArmor profile for it is difficult due to its complexity.

My Thoughts

All of these options have trade-offs between security, performance, complexity, and flexibility.

In my case, my priorities are:

  • Security: I do not trust the guest VMs. This means the guest should not connect to the host, and the host should not load the guest's disk image using a kernel module. While I could move the backup logic to a separate VM, this would add a lot of complexity.
  • Simplicity: I want a simple workflow that is easy to maintain and secure. I prefer to avoid writing complicated AppArmor profiles that are difficult to update.
  • Performance: I don't need the backup to be super fast, meanwhile I don't want to keep a VM shut down for too long.
  • Space Efficiency: I don't have a large amount of data, so disk space is not a major concern.
Considering these factors, I prefer backing up the entire disk image. Recall that my root filesystem is created from a Containerfile, and /etc is transient, so I primarily need to back up /var.

For now, I plan to use qcow2 images with compression. I will decompress the disk image and pipe the data to the backup software.

In the future, I might explore some optimizations:
  • Using a raw disk image on a ZFS host filesystem with compression and possibly deduplication enabled.
  • Taking a snapshot (either qcow2 or ZFS) and backing it up. This would allow me to restart the VM without waiting for the backup to finish.

Labels:

Comments

Post a Comment

Popular posts from this blog

A Rocky Migration: Moving from docker-compose to Podman and gVisor

I've been running a few containers for several years. They were all running under rootless Docker with a single user. Initially, I planned to  migrate the containers to VMs , but I couldn't get a stable workflow after about two months of effort. Later,  gVisor caught my attention , and I decided to migrate to Podman with gVisor instead. The new plan is to run each container with  --userns=auto  and use Quadlet for systemd integration. This approach provides better isolation and makes writing firewall rules easier. I'm now close to migrating all my containers. Here are a couple of rough edges I'd like to share. Network Layout I compared  various networking options  and spent a few hours trying the one-interface-per-group approach before giving up. I settled on a single macvlan network and decided to use static IP addresses for my containers. To prevent a randomly assigned IP address from conflicting with a predefined one, I allocated a large IP range for my ...
Post a Comment
Read More »

Exploring Immutable Distros and Declarative Management

My current server setup, based on Debian Stable and Docker, has served me reliably for years. It's stable, familiar, and gets the job done. However, an intriguing article I revisited recently about Fedora CoreOS, rpm-ostree, and OSTree native containers sparked my curiosity and sent me down a rabbit hole exploring alternative approaches to system management. Could there be a better way? Core Goals & Requirements Before diving into new technologies, I wanted to define what "better" means for my use case: The base operating system must update automatically and reliably. Hosted services (applications) should be updatable either automatically or manually, depending on the service. Configuration and data files need to be easy to modify, and crucially, automatically tracked and backed up. Current Setup: Debian Stable + Docker My current infrastructure consists of several servers, all running Debian Stable. System Updates are andled automatically via unattended-upgrades. Se...
Post a Comment
Read More »

mkosi: First Impressions

I stumbled upon the Gentoo wiki page for systemd-nspawn , which in turn led me to nspawn.org , mkosi , and later systemd-sysupdate . mkosi quickly caught my eye because it's almost exactly what I wanted to build myself, as mentioned in a previous post . So, I decided to spend my "sysadmin fun quota" on it. Overview mkosi is similar to docker build or podman build , but it's designed for creating full OS images. It focuses on development and testing. For example, much like nix-shell , mkosi can quickly launch a sandboxed shell with a specific distribution and selected packages installed. The systemd project itself uses mkosi for testing across different distros. The re-introduction article  is a great read. Speed Note that this is by no means a rigid benchmark. My setup is an SSD with LUKS and an ext4 filesystem (without reflink support). Building Container Images mkosi is pretty fast. A simple mkosi command creates a fresh Debian image. I used the --incrementa...
Post a Comment
Read More »