Dracut Emergency Shell /dev/disk/by-uuid/<Nr> does not exist

[pasco]

Hi 2gether

Some days ago I did a “yum update” on the CLI of my freepbx distro. The update failed and the machine freezed. So I tried a reboot. I’ve only got a “kernel panic”-screen.

I found out, there must have been something going wrong with a change from grub legacy to grub2 (?!). Actually grub2 was installed, but it was still starting from grub legacy. After some trying I finally could start into an old kernel. But I couldn’t let it stand like that, because something was wrong. I also recognized, the initramfs-file for the latest kernel (957) was missing, so I’ve generated it by using dracut.

So I’ve finally could switch to grub2. In the beginning the system starts but then, nearly the end of the start up process, I get a lot of “dracut-initqueue[250]: Warning: darcut-initqueue timeout - starting timeout scripts”. Then a “dracut-initqueue[250]: Warning: Could not boot.” followed by “dracut-initqueue[250]: Warning: /dev/disk/by-uuid/53925f99-d467-438f-aef4-eac70192eae5 does not exist”. Actually that is the UUID of md2, as blkid says.

I’ve searched now for two days on the internet and tried a lot of things. Nothing helped. I can’t start my freepbx-distro-machine anymore, because the older kernels doesn’t start anymore neither :-(. I still can get into the emergency mode and chroot /mnt/sysimage.

Maybe one of my two disks from the raid1 is damaged. I’ve found errors like that in dmesg:
[ 88.222563] EXT4-fs error (device md2): ext4_ext_find_extent:885: inode #63570614: comm sshd: pblk 254312474 bad header/extent: invalid extent entries - magic f30a, entries 190, max 340(340), depth 0(0)
[ 88.222755] EXT4-fs error (device md2): ext4_ext_remove_space:2976: inode #63570614: comm sshd: pblk 254312474 bad header/extent: invalid extent entries - magic f30a, entries 190, max 340(340), depth 0(0)
[ 88.222943] EXT4-fs error (device md2) in ext4_ext_truncate:4688: IO failure

Could this be the problem? Or is it somewhere else? Maybe a “wrong” initramfs-file? I’ve also had to “reassemble” md2, because one partition was missing (maybe because one of the two disks is faulty?). Even though it has resynced without problems? Why does it say, it can’t find the UUID of md2?

I really appreciate any hint/help because this is our productive machine in the office and I can’t get calls for the last two days…thanks so much in advance!

Cheers, Pasco

[pasco]

I think I’ve found the reason for the error message:

If I do a “blkid” in the emergency dracut shell it doesn’t show the raid1 arrays (md0, md1, md2). So it’s logical it can’t find the UIDD of md2.

Where could be the problem? In the initramfs / it’s compilation (because I had to do it “by hand” as yum crashed during update) or in the grub.cfg? Missing raid-drivers or activation or something like that?

Thanks for any help.
Pasco

[pasco]

Good news. I’ve found a solution, my freepbx box is up and running .

If someone else ever gets to this point with his/her freepbx-distro raid1 box, maybe my way can help. What I did:

Regenerate initramfs with dracut and some options:

dracut --mdadmconf --fstab --add="dm dmraid mdraid" --add-drivers="raid1" --force /boot/initramfs-3.10.0-957.10.1.el7.x86_64.img 3.10.0-957.10.1.el7.x86_64 -M

Check if there is a file “grub” in /etc/default with the following content:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed ‘s, release .*$,g’ /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT=“gfxterm”
GRUB_CMDLINE_LINUX=“rd.auto rd.auto=1 rd.luks=0 KEYBOARDTYPE=pc vconsole.keymap=us rd.locale.LANG=en_US.UTF-8 vconsole.font=latarcyrheb-sun16 crashkernel=auto rd.md.uuid=<UUID you’ve got out of mdadm --detail-dev-md2> rd.lvm=0 rd.md.uuid=<UUID you’ve got out of mdadm --detail-dev-md1> rd.dm=0 rhgb quiet biosdevname=0 net.ifnames=0 LANG=en_US.UTF-8”

GRUB_PRELOAD_MODULES=“mdraid1x”
GRUB_DISABLE_RECOVERY=“true”
GRUB_BACKGROUND=/boot/grub/splash.jpg

Then a
grub2-mkconfig -o /boot/grub2/grub.cfg

That was it.

Thanks for your attention :-).
Cheers, Pasco