Solving LVM Detection Failures in GRUB After a Force Shutdown

After a routine system update and an unfortunate hang that required a hard reset, my Linux machine refused to boot. Instead of the familiar login prompt, I was greeted by a cryptic GRUB error: error: no cryptodisk module can handle this device.

My setup uses LUKS2 + LVM. From the GRUB rescue shell, I could actually decrypt the LUKS container. But once decrypted, GRUB completely failed to detect any LVM volumes. It simply acted as if the LVM structure didn't exist.

Meanwhile, if I boot it from a Live Rescue USB, everything worked perfectly. I could open the LUKS container, and the volume group appeared immediately. Tools like vgck and pvck reported no issues.

After a length discussion with AI, eventually I found the magical commands:

vgcfgbackup -f lvm_backup.txt <vg_name>
vgcfgrestore -f lvm_backup.txt <vg_name>

After running these commands and rebooting, GRUB recognized the LVM volumes immediately, and I was back in my system.

Supposedly this forces rewriting the LVM metadata. Perhaps there were issues with the metadata, which were caused by the forced shutdown. The issues can be handled by LVM parser in Linux, but not by the limited implementation in GRUB.

Comments

Exploring Immutable Distros and Declarative Management

My current server setup, based on Debian Stable and Docker, has served me reliably for years. It's stable, familiar, and gets the job done. However, an intriguing article I revisited recently about Fedora CoreOS, rpm-ostree, and OSTree native containers sparked my curiosity and sent me down a rabbit hole exploring alternative approaches to system management. Could there be a better way? Core Goals & Requirements Before diving into new technologies, I wanted to define what "better" means for my use case: The base operating system must update automatically and reliably. Hosted services (applications) should be updatable either automatically or manually, depending on the service. Configuration and data files need to be easy to modify, and crucially, automatically tracked and backed up. Current Setup: Debian Stable + Docker My current infrastructure consists of several servers, all running Debian Stable. System Updates are andled automatically via unattended-upgrades. Se...

A Rocky Migration: Moving from docker-compose to Podman and gVisor

I've been running a few containers for several years. They were all running under rootless Docker with a single user. Initially, I planned to migrate the containers to VMs , but I couldn't get a stable workflow after about two months of effort. Later, gVisor caught my attention , and I decided to migrate to Podman with gVisor instead. The new plan is to run each container with --userns=auto and use Quadlet for systemd integration. This approach provides better isolation and makes writing firewall rules easier. I'm now close to migrating all my containers. Here are a couple of rough edges I'd like to share. Network Layout I compared various networking options and spent a few hours trying the one-interface-per-group approach before giving up. I settled on a single macvlan network and decided to use static IP addresses for my containers. To prevent a randomly assigned IP address from conflicting with a predefined one, I allocated a large IP range for my ...

Determine Perspective Lines With Off-page Vanishing Point

In perspective drawing, a vanishing point represents a group of parallel lines, in other words, a direction. For any point on the paper, if we want a line towards the same direction (in the 3d space), we simply draw a line through it and the vanishing point. But sometimes the vanishing point is too far away, such that it is outside the paper/canvas. In this example, we have a point P and two perspective lines L1 and L2. The vanishing point VP is naturally the intersection of L1 and L2. The task is to draw a line through P and VP, without having VP on the paper. I am aware of a few traditional solutions: 1. Use extra pieces of paper such that we can extend L1 and L2 until we see VP. 2. Draw everything in a smaller scale, such that we can see both P and VP on the paper. Draw the line and scale everything back. 3. Draw a perspective grid using the Brewer Method. #1 and #2 might be quite practical. #3 may not guarantee a solution, unless we can measure distances/p...

WangLu's Notes

Search This Blog