Skip to main content

Backing Up VM Disk Images

[Update] I managed to work out the AppArmor profile, and decided to go with guestmount for now.

I am setting up a maintenance pipeline for my virtual machines.

The pipeline has two main routines:

  1. The BACKUP routine: Every day, this routine shuts down each VM, backs up the data in /var, updates the VM's disk image if a new version is available, and then restarts it.
  2. The BUILD routine: Every week, this routine uses a special builder VM to create new disk images for all VMs.
There is a scheduling conflict with the builder VM: the BACKUP routine needs to shut it down, while the BUILD routine needs it running. To resolve this, I merged both into a single set of systemd services that runs daily. The BUILD routine starts automatically when the builder VM starts, at the end of the BACKUP routine. The builder VM's systemd unit has an ExecCondition= property, which is skipped 6 days a week.

Surprisingly, the most difficult part of this pipeline was not the scheduling, but the backup process itself.

There are two general approaches to backing up a VM: backing up the entire disk image or backing up the files directly from the filesystem.

Backing Up Disk Images

Backing up a disk image is straightforward because disk images are just regular files. However, this method is often inefficient:
  • Deduplication may not work well if the disk image is compressed.
  • Incremental backups are not natively supported.
  • The entire disk image must be backed up, including unused and deleted data.
  • You cannot easily choose which specific files to include or exclude from the backup.
There are some ways to improve this approach:
  • For deduplication: I can decompress the disk image and pipe the data stream directly to the backup software without saving the decompressed file.
  • For incremental backups: I can create snapshots and back up only the differences. I would also need to regularly merge the snapshots.
    • QEMU supports incremental backup for a running VM. Relavent article: 1, 2.
  • To reduce backup size: I can defragment, shrink, wipe, and sparsify the disk image (for example, with virt-sparsify) before backing it up.
  • To exclude specific files: I can put the files or directories that I don't want to back up on a separate disk image.

Backing Up Filesystems

One can back up files from inside the VM. It is also possible to mount the disk image from the host system when the VM is shut down.

Raw disk images can be mounted directly using loop devices.

Qcow2 images have several options:
  • `qemu-nbd` can expose an image as a block device (e.g., /dev/nbd*), which can then be mounted. This requires the nbd kernel module and root access. The qemu-nbd man page warns that this may not be suitable for untrusted guests. To back up multiple VMs, I would also need a way to find an available /dev/nbd* device.
    • The block device can be exported without root, there are tools like `qemu-nbd`, `nbdfuse` and `qemu-storage-daemon`. This article is worth reading.
    • `qemu-nbd` also supports exposing an internal snapshot of a qcow2 image.
  • `guestmount` can mount a disk image without needing root. It uses QEMU and FUSE. While it works, it can be slow, and creating a secure AppArmor profile for it is difficult due to its complexity.

My Thoughts

All of these options have trade-offs between security, performance, complexity, and flexibility.

In my case, my priorities are:

  • Security: I do not trust the guest VMs. This means the guest should not connect to the host, and the host should not load the guest's disk image using a kernel module. While I could move the backup logic to a separate VM, this would add a lot of complexity.
  • Simplicity: I want a simple workflow that is easy to maintain and secure. I prefer to avoid writing complicated AppArmor profiles that are difficult to update.
  • Performance: I don't need the backup to be super fast, meanwhile I don't want to keep a VM shut down for too long.
  • Space Efficiency: I don't have a large amount of data, so disk space is not a major concern.
Considering these factors, I prefer backing up the entire disk image. Recall that my root filesystem is created from a Containerfile, and /etc is transient, so I primarily need to back up /var.

For now, I plan to use qcow2 images with compression. I will decompress the disk image and pipe the data to the backup software.

In the future, I might explore some optimizations:
  • Using a raw disk image on a ZFS host filesystem with compression and possibly deduplication enabled.
  • Taking a snapshot (either qcow2 or ZFS) and backing it up. This would allow me to restart the VM without waiting for the backup to finish.

Labels:

Comments

Post a Comment

Popular posts from this blog

Determine Perspective Lines With Off-page Vanishing Point

In perspective drawing, a vanishing point represents a group of parallel lines, in other words, a direction. For any point on the paper, if we want a line towards the same direction (in the 3d space), we simply draw a line through it and the vanishing point. But sometimes the vanishing point is too far away, such that it is outside the paper/canvas. In this example, we have a point P and two perspective lines L1 and L2. The vanishing point VP is naturally the intersection of L1 and L2. The task is to draw a line through P and VP, without having VP on the paper. I am aware of a few traditional solutions: 1. Use extra pieces of paper such that we can extend L1 and L2 until we see VP. 2. Draw everything in a smaller scale, such that we can see both P and VP on the paper. Draw the line and scale everything back. 3. Draw a perspective grid using the Brewer Method. #1 and #2 might be quite practical. #3 may not guarantee a solution, unless we can measure distances/p...
3 comments
Read More »

Qubes OS: First Impressions

A few days ago, while browsing security topics online, Qubes OS surfaced—whether via YouTube recommendations or search results, I can't recall precisely. Intrigued by its unique approach to security through compartmentalization, I delved into the documentation and watched some demos. My interest was piqued enough that I felt compelled to install it and give it a try firsthand. My overall first impression of Qubes OS is highly positive. Had I discovered it earlier, I might have reconsidered starting my hardware password manager project. Conceptually, Qubes OS is not much different from running a bunch of virtual machines simultaneously. However, its brilliance lies in the seamless desktop integration and the well-designed template system, making it far more user-friendly than a manual VM setup. I was particularly impressed by the concept of disposable VMs for temporary tasks and the clear separation of critical functions like networking (sys-net) and USB handling (sys-usb) into the...
Post a Comment
Read More »

ESP32S3: Flash Encryption and Secure Boot

Flash encryption and secure boot are useful security features for ESP32S3 chip. While not perfect, they definitely make it harder to extract the secrets in the chip. However, it is tricky to enable both features at the same time. The topic is actually discussed in the official documentation: ESP32S3 Security Features Security Features Enablement Workflows Especially, the second one mentioned it is recommended to enable flash encryption before secure boot. But I still find the documentation confusing. In the end I was able to successfully enable both, here's my findings. My Understanding After my adventure, here's what I think could have worked. WARNING, this is untested. Follow  Security Features Enablement Workflows : Burn all the keys, as long as their purpose eFuses and read/write protections Burn other security eFuses, but DO NOT burn ENABLE_SECURITY_DOWNLOAD in the middle, which is mentined at the end of the instruction for both flash encryption and secure boot. Burn...
Post a Comment
Read More »