Fix the dracut emergency shell on Linux after a VMware to Proxmox migration
A Linux VM that booted fine on VMware stops at the dracut emergency shell on its first boot on Proxmox VE, oVirt / OLVM or any KVM hypervisor when its initramfs was built only for the VMware hardware and carries no VirtIO storage driver. The kernel starts, cannot find the root file system and gives up. The disk is intact. The fix is to boot once from something the old initramfs can read, rebuild the initramfs with the VirtIO drivers and switch back.
Why it happens
The initramfs is the small image the kernel uses to find and mount the root file system. On RHEL, Oracle Linux, Rocky, AlmaLinux and SLES, dracut builds it in host-only mode by default: it includes only the drivers for the hardware the system had at that moment. On VMware that is the VMware Paravirtual or the LSI Logic controller. On KVM the disk hangs off VirtIO SCSI or VirtIO Block, the initramfs has neither virtio_scsi nor virtio_blk, and the root volume never appears.
The kernel is not the problem: every supported distribution ships the VirtIO drivers as modules. They are simply not inside the initramfs. That is why the fix is to rebuild it, not to install anything.
Debian and Ubuntu build a generic initramfs by default, so they usually boot. They fail the same way when the system was configured with MODULES=dep.
A second, unrelated cause looks similar on recent systems: the CPU model. RHEL 10 and Rocky 10 require the x86-64-v3 instruction set, and a VM created with an older virtual CPU model, such as x86-64-v2-AES on Proxmox, does not get past the start of the boot.
Fix it by hand
These steps are written for Proxmox VE and apply the same way to any KVM hypervisor. The rescue entry is the shortest path on the RHEL family. The installation ISO works on every distribution.
- 01Set a CPU model the system accepts
For RHEL 10, Rocky 10 and other systems built for x86-64-v3, set the CPU type of the VM to host, or to a model that includes x86-64-v3. On Proxmox it is the Type field under Processors.
- 02Boot the rescue entry
On the RHEL family, GRUB keeps an entry with rescue in its name. Its initramfs is generic and includes the VirtIO drivers, so it boots where the normal entry does not. Open the GRUB menu at startup and choose it.
- 03If there is no rescue entry, boot from the controller the old initramfs knows
If the VM used the VMware Paravirtual controller, set the SCSI controller of the VM to VMware PVSCSI and start it. The initramfs built on VMware carries that driver, so the root volume appears again. If it used another controller, go to the installation ISO step.
- 04Rebuild the initramfs
Once the system is up, rebuild the initramfs of every installed kernel with the VirtIO drivers included, and check that they are inside.
dracut -f --regenerate-all --add-drivers "virtio_blk virtio_scsi virtio_pci virtio_net" lsinitrd | grep virtio - 05On Debian and Ubuntu
Add the VirtIO modules to the initramfs configuration and rebuild it for every kernel.
printf 'virtio_blk\nvirtio_scsi\nvirtio_pci\nvirtio_net\n' | sudo tee -a /etc/initramfs-tools/modules sudo update-initramfs -u -k all - 06Switch back to VirtIO SCSI
Shut down, set the SCSI controller to VirtIO SCSI and start the VM from its normal GRUB entry. If /etc/fstab mounts by device name instead of UUID or LVM name, fix it first: with VirtIO Block the disks are called vda, vdb and so on.
- 07If nothing boots, use the installation ISO
Boot the installation ISO of the same distribution in rescue mode, let it mount the installed system and rebuild the initramfs from inside it. On the RHEL family the installed system is mounted under /mnt/sysimage or /mnt/sysroot: the rescue screen says which.
chroot /mnt/sysimage dracut -f --regenerate-all --add-drivers "virtio_blk virtio_scsi virtio_pci virtio_net" exit
How Moov avoids it
Moov rebuilds the boot image of the VM while it prepares it for KVM, offline and against the mounted disk, so the first boot on the target already finds its root volume. There is no rescue entry to choose and no second boot.
Download the appliance
- The rebuilt boot image is checked before it replaces the original. If something essential is missing, the migration stops with the reason and the original disk is left untouched.
- It covers the disk layouts found in the field: LVM spread over several disks, /boot on a different disk than the root, and the LVM devices file of EL9 and later.
- Validated on OLVM and HPE VM Essentials with Oracle Linux 8, 9 and 10 on BIOS and UEFI, RHEL, Rocky, SLES and Ubuntu.
- On Proxmox VE the VM gets the host CPU model by default, so RHEL 10 and Rocky 10 boot. You can change it per VM if your cluster is heterogeneous.
- The network is kept too: bonds, bridges, VLANs and aliases with NetworkManager, netplan, systemd-networkd and SUSE, and adapters named eth0, eth1 and so on.
- After the migration Moov powers the VM on and waits for the QEMU guest agent. The wave reports verified, inconclusive or failed instead of turning green just because the disks copied.
Frequently asked questions
- What is the dracut emergency shell?
- It is the minimal shell the initramfs opens when it cannot find or mount the root file system. After a change of hypervisor it almost always means that the initramfs lacks the driver for the new disk controller, not that the disk is damaged.
- Did I lose my data?
- No. The disk and its LVM volumes are intact. The initramfs simply cannot reach them through the VirtIO controller until it is rebuilt.
- I migrated with an earlier version of Moov and the VM stops at the emergency shell. What do I do?
- Oracle Linux and RHEL VMs migrated with v1.0.39 through v1.0.42 need dracut -f --regenerate-all run once inside the VM. Boot the rescue entry to get in, run it and restart. It is fixed since v1.0.43.
- Does it happen on Ubuntu and Debian?
- Rarely, because their initramfs is generic by default and already includes the VirtIO drivers. It happens when the system was set to MODULES=dep, and the fix is update-initramfs after adding the VirtIO modules.
- Why does RHEL 10 or Rocky 10 not boot even with the right initramfs?
- Because they require the x86-64-v3 instruction set. A VM with an older virtual CPU model, such as x86-64-v2-AES, stops at the very start of the boot. Set the CPU type to host or to a model with x86-64-v3.