Debugging Grub Boot Problems

I just had a remote server lose its grub.cfg file, which caused it to boot to a grub prompt and go no further. As I was fixing this issue I found some useful commands along the way. I want to record these here in case I ever need to do this again.

Listing disk contents from grub prompt

To list the content of a partition do:

grub> set pager=1
grub> ls
(hd0) (hd0,gpt1) (hd0,gpt2)
grub> ls (hd0,gpt2)/
lost+found/ bin/ boot/ cdrom/ dev/ etc/ home/  lib/
lib64/ media/ mnt/ opt/ proc/ root/ run/ sbin/ 
srv/ sys/ tmp/ usr/ var/
grub> cat (hd0,gpt2)/boot/grub2/grub.cfg
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
set pager=1
...

Booting from the grub prompt

Manually entering the boot commands that are in the grub.cfg file will boot the system:

grub> load_video
grub> set gfxpayload=keep
grub> insmod gzio
grub> insmod part_gpt
grub> insmod btrfs
grub> set root='hd0,gpt2'
grub> linuxefi /boot/vmlinuz-5.19.8-100.fc35.x86_64 root=UUID=962cdc00-b466-49af-89ec-74f0f41c1429 ro rd.auto  crashkernel=auto vga=normal nomodeset LANG=en_US.UTF-8
grub> initrdefi /boot/initramfs-5.19.8-100.fc35.x86_64.img
brub> boot

Using chroot in rescue mode recovery

I could have used the chroot command to configure the disks to allow me to use the grub tools as normal after booting into rescue mode.

To do this mount the system’s root partition somewhere, and its boot partition if that’s separate. Bind mount the rescue system’s run-time directories into that filesystem as well. Then chroot to that filesystem:

$ sudo mount /dev/sda2 /mnt

$ sudo mkdir /mnt/boot               <====  if separate boot partition
$ sudo mount /dev/sdXX /mnt/boot     <====

$ sudo mount --bind /dev /mnt/dev
$ sudo mount --bind /dev/pts /mnt/dev/pts
$ sudo mount --bind /proc /mnt/proc
$ sudo mount --bind /sys /mnt/sys

$ sudo chroot /mnt

Fixing the actual problem

The problem was that the /boot/efi/EFI/fedora/grub.cfg file was out of date, pointing at boot image files that no longer existed. Fedora normally puts grub commands in this copy of the grub.cfg file that simply redirect the boot process to the /boot/grub2/grub.cfg file, which contains the actual boot menu:

search --no-floppy --fs-uuid --set=dev 962cdc00-b466-49af-89ec-74f0f41c1429
set prefix=($dev)/boot/grub2
export $prefix
configfile $prefix/grub.cfg