Moovv1.0.44
Moov · How-to guides

Fix the dracut emergency shell on Linux after a VMware to Proxmox migration

A Linux VM that booted fine on VMware stops at the dracut emergency shell on its first boot on Proxmox VE, oVirt / OLVM or any KVM hypervisor when its initramfs was built only for the VMware hardware and carries no VirtIO storage driver. The kernel starts, cannot find the root file system and gives up. The disk is intact. The fix is to boot once from something the old initramfs can read, rebuild the initramfs with the VirtIO drivers and switch back.

Why it happens

The initramfs is the small image the kernel uses to find and mount the root file system. On RHEL, Oracle Linux, Rocky, AlmaLinux and SLES, dracut builds it in host-only mode by default: it includes only the drivers for the hardware the system had at that moment. On VMware that is the VMware Paravirtual or the LSI Logic controller. On KVM the disk hangs off VirtIO SCSI or VirtIO Block, the initramfs has neither virtio_scsi nor virtio_blk, and the root volume never appears.

The kernel is not the problem: every supported distribution ships the VirtIO drivers as modules. They are simply not inside the initramfs. That is why the fix is to rebuild it, not to install anything.

Debian and Ubuntu build a generic initramfs by default, so they usually boot. They fail the same way when the system was configured with MODULES=dep.

A second, unrelated cause looks similar on recent systems: the CPU model. RHEL 10 and Rocky 10 require the x86-64-v3 instruction set, and a VM created with an older virtual CPU model, such as x86-64-v2-AES on Proxmox, does not get past the start of the boot.

Fix it by hand

These steps are written for Proxmox VE and apply the same way to any KVM hypervisor. The rescue entry is the shortest path on the RHEL family. The installation ISO works on every distribution.

  1. 01
    Set a CPU model the system accepts

    For RHEL 10, Rocky 10 and other systems built for x86-64-v3, set the CPU type of the VM to host, or to a model that includes x86-64-v3. On Proxmox it is the Type field under Processors.

  2. 02
    Boot the rescue entry

    On the RHEL family, GRUB keeps an entry with rescue in its name. Its initramfs is generic and includes the VirtIO drivers, so it boots where the normal entry does not. Open the GRUB menu at startup and choose it.

  3. 03
    If there is no rescue entry, boot from the controller the old initramfs knows

    If the VM used the VMware Paravirtual controller, set the SCSI controller of the VM to VMware PVSCSI and start it. The initramfs built on VMware carries that driver, so the root volume appears again. If it used another controller, go to the installation ISO step.

  4. 04
    Rebuild the initramfs

    Once the system is up, rebuild the initramfs of every installed kernel with the VirtIO drivers included, and check that they are inside.

    dracut -f --regenerate-all --add-drivers "virtio_blk virtio_scsi virtio_pci virtio_net"
    lsinitrd | grep virtio
  5. 05
    On Debian and Ubuntu

    Add the VirtIO modules to the initramfs configuration and rebuild it for every kernel.

    printf 'virtio_blk\nvirtio_scsi\nvirtio_pci\nvirtio_net\n' | sudo tee -a /etc/initramfs-tools/modules
    sudo update-initramfs -u -k all
  6. 06
    Switch back to VirtIO SCSI

    Shut down, set the SCSI controller to VirtIO SCSI and start the VM from its normal GRUB entry. If /etc/fstab mounts by device name instead of UUID or LVM name, fix it first: with VirtIO Block the disks are called vda, vdb and so on.

  7. 07
    If nothing boots, use the installation ISO

    Boot the installation ISO of the same distribution in rescue mode, let it mount the installed system and rebuild the initramfs from inside it. On the RHEL family the installed system is mounted under /mnt/sysimage or /mnt/sysroot: the rescue screen says which.

    chroot /mnt/sysimage
    dracut -f --regenerate-all --add-drivers "virtio_blk virtio_scsi virtio_pci virtio_net"
    exit
Moov v1.0.44

How Moov avoids it

Moov rebuilds the boot image of the VM while it prepares it for KVM, offline and against the mounted disk, so the first boot on the target already finds its root volume. There is no rescue entry to choose and no second boot.

  • The rebuilt boot image is checked before it replaces the original. If something essential is missing, the migration stops with the reason and the original disk is left untouched.
  • It covers the disk layouts found in the field: LVM spread over several disks, /boot on a different disk than the root, and the LVM devices file of EL9 and later.
  • Validated on OLVM and HPE VM Essentials with Oracle Linux 8, 9 and 10 on BIOS and UEFI, RHEL, Rocky, SLES and Ubuntu.
  • On Proxmox VE the VM gets the host CPU model by default, so RHEL 10 and Rocky 10 boot. You can change it per VM if your cluster is heterogeneous.
  • The network is kept too: bonds, bridges, VLANs and aliases with NetworkManager, netplan, systemd-networkd and SUSE, and adapters named eth0, eth1 and so on.
  • After the migration Moov powers the VM on and waits for the QEMU guest agent. The wave reports verified, inconclusive or failed instead of turning green just because the disks copied.

Frequently asked questions

What is the dracut emergency shell?
It is the minimal shell the initramfs opens when it cannot find or mount the root file system. After a change of hypervisor it almost always means that the initramfs lacks the driver for the new disk controller, not that the disk is damaged.
Did I lose my data?
No. The disk and its LVM volumes are intact. The initramfs simply cannot reach them through the VirtIO controller until it is rebuilt.
I migrated with an earlier version of Moov and the VM stops at the emergency shell. What do I do?
Oracle Linux and RHEL VMs migrated with v1.0.39 through v1.0.42 need dracut -f --regenerate-all run once inside the VM. Boot the rescue entry to get in, run it and restart. It is fixed since v1.0.43.
Does it happen on Ubuntu and Debian?
Rarely, because their initramfs is generic by default and already includes the VirtIO drivers. It happens when the system was set to MODULES=dep, and the fix is update-initramfs after adding the VirtIO modules.
Why does RHEL 10 or Rocky 10 not boot even with the right initramfs?
Because they require the x86-64-v3 instruction set. A VM with an older virtual CPU model, such as x86-64-v2-AES, stops at the very start of the boot. Set the CPU type to host or to a model with x86-64-v3.