Grub Recovery: Importing ZFS dataset with altroot for grub recovery loses original mountpoint paths in chroot

[berglh]

Hello,

I've just embarked on setting up a UEFI home machine to boot Ubuntu 25.04 with ZFS root. I've had experience on BIOS/legacy boot before since 19.10 that has been rock solid.

I am following the 22.04 guide with the appropriate changes for 25.04. I also used a simplified structure that I had been using on a different server (with BIOS/legacy booting), rather than rpool/ROOT/dataset/bpool/BOOT/dataset pattern in the guide.

Datasets

ℹ️ boot, ubuntu, ubuntu/var containers have canmount=off

NAME               USED  AVAIL  REFER  MOUNTPOINT
boot               170M  1.58G    96K  /boot
boot/boot          169M  1.58G  90.9M  /boot
ubuntu            1.23G   883G    96K  /
ubuntu/home        280K   883G    96K  /home
ubuntu/home/root   184K   883G   184K  /root
ubuntu/root       1015M   883G  1013M  /mnt
ubuntu/tmp         124K   883G   124K  legacy
ubuntu/var         243M   883G    96K  /var
ubuntu/var/cache   129M   883G   129M  /var/cache
ubuntu/var/lib     112M   883G   112M  /var/lib
ubuntu/var/log    2.27M   883G  2.27M  legacy
ubuntu/var/mail     96K   883G    96K  /var/mail
ubuntu/var/snap     96K   883G    96K  /var/snap
ubuntu/var/spool   112K   883G   112K  legacy

EFI mounts

/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/nvme0n1p1 on /boot/grub type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

The mistake

I made a bit of a mistake following an old markdown guide I wrote for Ubuntu 19.10 by marking my boot (/boot) ZFS dataset as legacy and trying to mount it with fstab. This resulted in the system booting, starting grub, but entering maintenance recovery mode with a hard lock (unable to login to troubleshoot). Editing fstab and commenting out the /boot mount got it starting correctly, but then the boot dataset wasn't mounted.

I then attempted to boot off the Live USB key again, then imported my zpools with altroot set to /mnt.

zpool import -f -N -R /mnt ubuntu
zpool import -f -N -R /mnt root

I then mounted all my datasets, FAT32 EFI partition to /boot/efi, Grub bind /boot/efi/grub->/boot/grub mount. When I zfs get mountpoint at this stage, I see all my datasets are prefixed with the altroot path.

I then chroot in after binding /dev, /proc, /sys. When I zfs get mountpoint at this stage, all my datasets are still referencing the /mnt prefixed paths, the same as outside of chroot.

When I try to update-grub after setting boot/boot to a non-legacy mount and commenting out the /boot mount in /etc/fstab I get the following error:

 update-grub
Sourcing file /etc/default/grub'
Generating grub configuration file ...
Warning: didn't find any valid initrd or kernel.
filesystem 'boot/boot' cannot be mounted using 'mount'.
Use 'zfs set mountpoint=legacy' or 'zfs mount boot/boot'.
See zfs(8) for more information.
Warning: didn't find any valid initrd or kernel.
Adding boot menu entry for UEFI Firmware Settings ...
done

I've tried to adjust my /etc/zfs/zfs-list.cache files removing the /mnt prefix as per the guide, but nothing I seem to do allows me to successfully update-grub.

ℹ️ I seem to remember when I created the ZFS pools and datasets in the first instance with altroot set, that these paths in chroot were referencing the expect non-/mnt prefixed paths. Maybe I was mistaken.

GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=3
GRUB_RECORDFAIL_TIMEOUT=5
GRUB_DISTRIBUTOR=`( . /etc/os-release; echo ${NAME:-Ubuntu} ) 2>/dev/null || echo Ubuntu`
GRUB_CMDLINE_LINUX_DEFAULT="init_on_alloc=0"
GRUB_CMDLINE_LINUX=""
GRUB_TERMINAL=console

I can blow everything away and start again, and that will probably fix my issue, but I am concerned that I am unable update-grub from a Live USB boot in the event I require future troubleshooting. Wondering if I'm missing something simple that will allow ZFS to see the correct mount points and allow grub to succeed. Obviously, I've stepped away from some of the core strategies in the guide and it's not clear where I should be focusing my efforts.

I will also add I'd prefer to stay using grub as the bootloader, so avoiding using ZFSBootManager etc.

Any help would be greatly appreciated 🤗

[berglh]

Ok, so - I think some of my issue was setting some of my rpool datasets in /var as legacy mount type and specifying them in fstab, as was the case some time ago.

I reinstalled again, following the 22.04 guide step-by-step with some changes for 25.04 and everything was working as expected. I documented each command in a Gist Minimal ZFS Root on Ubuntu 25.04 for troubleshooting but it is booting via grub now. When I get back from holiday I will try to update grub from the Live USB environment again, this wasn't working when I created this discussion.