Why Generic Linux RAID 1 Scripts Fail on the HP MicroServer Gen8 (And How to Do It Right)

By | September 28, 2026
0 Comment

In our previous guide (Zero-Downtime Linux OS Mirroring: Converting a Single-Disk Server to Live High-Availability RAID 1), we demonstrated how to convert a running single-disk Linux server to a software RAID 1 mirror using mdadm, pvmove, and grub2-install.

On standard modern UEFI systems, that generic script runs smoothly. But when applied to an HP ProLiant MicroServer Gen8 (or similar legacy enterprise hardware running modern Fedora / RHEL), subtle hardware architecture quirks and modern Linux systemd/LVM subsystem changes can lead to boot failure, emergency mode drops, or console lockups upon restart.

In this deep-dive companion post, we dissect why generic RAID conversion scripts fail on HP Gen8 hardware, explain the critical difference between soft reboots vs. cold hardware power cycles, examine how to recover without locking the emergency console, and provide the battle-tested, Gen8-specific script to execute this migration cleanly from the beginning.

🏗 The Architecture & Hardware Reality

The HP ProLiant MicroServer Gen8 is a legendary homelab workhorse, but its storage and boot architecture combines Legacy BIOS (non-UEFI), the Intel C204 chipset, an HP Dynamic Smart Array B120i / LSI SAS controller, and an iLO 4 management processor.

When modern Linux (Fedora 40+) meets this legacy enterprise boot architecture, distinct hardware and software traps emerge:

⚠️ The 7 Traps That Break Generic RAID 1 on HP Gen8

1. The USB 3.0 Boot-Keyboard Blindness

  • The Symptom: When trying to access the BIOS setup (F9) or Boot Menu (F11), keyboard presses are completely ignored, and the screen stays blank.
  • The Root Cause: The rear Blue USB 3.0 ports on the Gen8 are driven by a third-party Renesas controller that is not initialized by the Legacy BIOS during POST.
  • The Rule: Keyboards and bootable Live USB sticks must only be plugged into the Front USB ports or the Rear Black USB 2.0 ports (which connect directly to the Intel C204 PCH).

2. Option ROM Interrupt Interception

  • The Symptom: Function keys (F9, F11) fail even on USB 2.0 ports if pressed slightly too late.
  • The Root Cause: The HP Smart Array B120i and LSI SAS Option ROMs hijack hardware interrupts during drive probing.
  • The Solution: Use HP iLO 4 (Administration → Boot Order → One-Time Boot: USB) to force the next boot directly into the USB environment from the web UI, bypassing keyboard timing entirely.

3. MBR Partition Active Flag Stripping

  • The Symptom: The BIOS finishes POST, initializes disks, and then sits indefinitely at a blank screen with a blinking cursor without loading GRUB.
  • The Root Cause: In standard MBR (DOS) partition tables, setting a partition flag to raid on in parted changes the type ID to 0xfd but can strip the legacy 0x80 Active Bootable Flag (*) on Partition 1. The HP Legacy BIOS strictly requires this flag to execute the Master Boot Record.

4. Missing rd.md.uuid Kernel Parameters

  • The Symptom: GRUB Stage 1 loads, but the kernel drops to an emergency shell with Warning: /dev/fedora00/root does not exist.
  • The Root Cause: In modern Fedora, dracut initramfs does not automatically assemble non-standard RAID arrays unless their UUIDs are explicitly declared in the kernel command line.
  • The Required Format:GRUB_CMDLINE_LINUX="rd.md.uuid=<OS_RAID_UUID> rd.lvm.lv=fedora00/root rd.md.uuid=<BOOT_RAID_UUID> rhgb quiet"

5. Fedora LVM Device Whitelisting (system.devices)

  • The Symptom: Dracut mounts /, but systemd drops into emergency mode with:fsconfig() failed: /dev/mapper/fedora00-home: Can't lookup blockdevRunning vgs, pvs, lvs returns completely blank output.
  • The Root Cause: Modern LVM enables strict device whitelisting (/etc/lvm/devices/system.devices). When LVM data is moved from a raw partition (sda2) to an mdadm device (/dev/md11), LVM ignores the new array because its hardware ID changed.
  • The Fix: Delete or update the whitelist file (rm -f /etc/lvm/devices/system.devices), which safely causes LVM to revert to standard full-device scanning across all block devices.

6. The Soft/Warm Reboot Trap vs. Cold Hardware Power Cycle

  • The Symptom: The server performs a live migration, but issuing systemctl reboot results in degraded drive enumeration, missing auxiliary block devices, or a timeout in systemd.
  • The Root Cause:
    • On a soft/warm restart, the Intel C204 / B120i SATA controller and Option ROM do not power-cycle the SATA physical lanes (PHY). The controller can report stale in-memory geometry or experience spin-up lag on secondary drives.
    • USB storage controllers also retain power, leading to asynchronous device discovery where /etc/fstab attempts to mount auxiliary drives before udev creates their device nodes.
  • The Rule: After performing an in-flight live storage migration on enterprise hardware, always perform a full Cold Power Cycle / Hard Reset (via iLO 4 Power Management -> Reset or physical power button) rather than a software warm reboot.

7. The Systemd Emergency Shell Console Lockup

  • The Symptom: If the server drops into emergency mode and you run vgimportdevices or complex LVM commands, the keyboard and console freeze solid.
  • The Root Cause: In systemd emergency mode:
    1. The root filesystem / is mounted Read-Only (ro).
    2. Background daemons (systemd-udevd, dmeventd) are suspended.
    3. When LVM commands attempt to write locks or communicate over IPC sockets with udev, they block indefinitely on the unserviced socket, freezing the console.
  • The Rule: In emergency mode, never run commands that depend on running background daemons. Simply remount / as rw, remove /etc/lvm/devices/system.devices, run vgchange -ay (which activates LVs via direct kernel ioctls without daemon dependencies), and run systemctl default.

🛡 Mandatory /etc/fstab Hardening: The nofail Rule

On multi-disk storage servers and homelabs, never leave secondary, backup, or external storage mounts with standard blocking mount options.

If a USB drive or secondary storage pool is slow to enumerate during boot, systemd will block the entire boot pipeline and dump the machine into emergency mode.

Ensure all non-root mounts in /etc/fstab include nofail:

# Root and Boot (Essential)
/dev/mapper/fedora00-root   /               ext4    defaults        1 1
UUID=&lt;BOOT_ARRAY_UUID>      /boot           ext4    defaults        1 2
/dev/mapper/fedora00-home   /home           ext4    defaults        1 2
/dev/mapper/fedora00-swap   swap            swap    defaults        0 0

# Secondary Storage Pools &amp; USB Drives (Hardened with nofail)
/dev/mapper/mypassport_luks /mnt/backup_3   ext4    rw,nofail       0 0
UUID=48d99476-8c18-49fb... /mnt/docker_ssd xfs     defaults,noatime,nofail 0 0
UUID=1e4b080a-1350-415b... /mnt/storage_1tb xfs    defaults,noatime,nofail 0 0

🛠 The Recovery Procedure (Fixing a Broken Boot via Live USB)

If your server is currently stuck after a failed RAID migration, do not panic—your data is intact on the SSDs. Follow this exact sequence in your Fedora Live USB terminal to restore bootability:

# ==============================================================================
# 1. Assemble Discovered Arrays &amp; Activate LVM
# ==============================================================================
sudo bash -c "mdadm --examine --scan >> /etc/mdadm.conf"
sudo mdadm --assemble --scan
sudo vgscan --devicesfile ""
sudo vgchange -ay --devicesfile ""

# ==============================================================================
# 2. Mount System into /mnt for Chroot
# ==============================================================================
sudo mount /dev/mapper/fedora00-root /mnt
sudo mount /dev/disk/by-label/boot_raid /mnt/boot 2>/dev/null || sudo mount /dev/md10 /mnt/boot

for d in /dev /dev/pts /proc /sys /run; do sudo mount --bind $d /mnt$d; done

# ==============================================================================
# 3. Clean Fedora LVM Whitelist
# ==============================================================================
# Remove restrictive devices file so LVM scans all arrays cleanly
sudo rm -f /mnt/etc/lvm/devices/system.devices

# ==============================================================================
# 4. Extract UUIDs and Configure /etc/mdadm.conf &amp; GRUB
# ==============================================================================
MD_BOOT_UUID=$(grep -E "/dev/md/?10" /etc/mdadm.conf | grep -o 'UUID=[^ ]*' | cut -d= -f2 | tr -d '"')
MD_OS_UUID=$(grep -E "/dev/md/?11" /etc/mdadm.conf | grep -o 'UUID=[^ ]*' | cut -d= -f2 | tr -d '"')

# Copy verified mdadm.conf to target system
sudo cp /etc/mdadm.conf /mnt/etc/mdadm.conf

# Inject explicit array UUIDs into kernel command line
sudo sed -i "s|^GRUB_CMDLINE_LINUX=.*|GRUB_CMDLINE_LINUX=\"rd.md.uuid=$MD_OS_UUID rd.lvm.lv=fedora00/root rd.md.uuid=$MD_BOOT_UUID rhgb quiet\"|" /mnt/etc/default/grub

# Generate fresh GRUB configuration
sudo chroot /mnt grub2-mkconfig -o /boot/grub2/grub.cfg

# ==============================================================================
# 5. Install GRUB to BOTH Physical SSDs with Embedded Modules
# ==============================================================================
DISKA=$(lsblk -d -no NAME,SERIAL | grep "&lt;SSD_A_SERIAL>" | awk '{print $1}')
DISKB=$(lsblk -d -no NAME,SERIAL | grep "&lt;SSD_B_SERIAL>" | awk '{print $1}')

sudo chroot /mnt grub2-install --modules="part_msdos mdraid1x diskfilter ext2 lvm biosdisk" "/dev/$DISKA"
sudo chroot /mnt grub2-install --modules="part_msdos mdraid1x diskfilter ext2 lvm biosdisk" "/dev/$DISKB"

# ==============================================================================
# 6. Rebuild Initramfs Images
# ==============================================================================
sudo chroot /mnt dracut -f --regenerate-all --add "mdraid lvm"

sync
# Perform a full hardware reset
sudo reboot

📜 The Bulletproof HP Gen8 Conversion Script (From Scratch)

If you are performing this migration on an HP MicroServer Gen8 from scratch, this comprehensive script integrates all hardware, software, and bootloader safeguards:

#!/bin/bash
# ==============================================================================
# Script: setup_hp_gen8_raid1.sh
# Target: HP ProLiant MicroServer Gen8 (Legacy BIOS / Fedora 40-44 / LVM)
# ==============================================================================
set -euo pipefail

LOGFILE="/var/log/hp_gen8_raid1_$(date +%Y%m%d_%H%M%S).log"
exec > >(tee -a "$LOGFILE") 2>&amp;1

echo "=========================================================="
echo "  HP MicroServer Gen8 High-Availability OS RAID 1 Setup   "
echo "=========================================================="

if [ "$EUID" -ne 0 ]; then
  echo "[-] ERROR: Must be run as root: sudo bash $0"
  exit 1
fi

DISK_SRC="/dev/sda"
DISK_DST="/dev/sdb"

# 1. Pre-flight Safety Backup
BACKUP_DIR="/root/os_pre_raid_backup_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"
cp -a /etc/fstab /etc/default/grub "$BACKUP_DIR/" 2>/dev/null || true
cp -a /etc/mdadm.conf "$BACKUP_DIR/" 2>/dev/null || true
rsync -aHAX /boot/ "$BACKUP_DIR/boot_backup/"
echo "[+] Safety backup created in $BACKUP_DIR"

# 2. Swap Reduction (Avoids pvmove metadata extent shortfall)
echo "[+] Adjusting swap to prevent LVM extent shortfall..."
swapoff /dev/mapper/fedora00-swap 2>/dev/null || true
wipefs -af /dev/mapper/fedora00-swap 2>/dev/null || true
lvresize -y -L 6G /dev/mapper/fedora00-swap
mkswap /dev/mapper/fedora00-swap
swapon /dev/mapper/fedora00-swap

# 3. Partition Target SSD with Active Boot Flag (MBR Required on Gen8)
echo "[+] Partitioning target drive ($DISK_DST)..."
for p in $(lsblk -lno NAME "$DISK_DST"); do umount -f "/dev/$p" 2>/dev/null || true; done
wipefs -af "$DISK_DST"
dd if=/dev/zero of="$DISK_DST" bs=1M count=20 conv=fsync status=none

parted -s "$DISK_DST" mklabel msdos
parted -s -a optimal "$DISK_DST" mkpart primary 2048s 2099199s
parted -s -a optimal "$DISK_DST" mkpart primary 2099200s 468861951s
parted -s "$DISK_DST" set 1 boot on
partprobe "$DISK_DST"
udevadm settle

# 4. Initialize Degraded RAID 1 Arrays
echo "[+] Initializing RAID 1 arrays on $DISK_DST..."
mdadm --zero-superblock --force "${DISK_DST}1" 2>/dev/null || true
mdadm --zero-superblock --force "${DISK_DST}2" 2>/dev/null || true

# Note: metadata=1.0 puts superblock at end of partition, readable by Legacy BIOS
mdadm --create /dev/md10 --level=1 --raid-devices=2 missing "${DISK_DST}1" --metadata=1.0 --force
mdadm --create /dev/md11 --level=1 --raid-devices=2 missing "${DISK_DST}2" --metadata=1.2 --force
udevadm settle

# 5. Format and Sync /boot
echo "[+] Syncing /boot..."
mkfs.ext4 -F -L boot_raid /dev/md10
mkdir -p /mnt/new_boot
mount /dev/md10 /mnt/new_boot
rsync -aHAX --delete /boot/ /mnt/new_boot/
umount /mnt/new_boot

# 6. Live LVM Migration &amp; Whitelist Removal
echo "[+] Migrating LVM Volume Group..."
pvcreate -ff -y /dev/md11
vgextend fedora00 /dev/md11

# Remove restrictive Fedora LVM devices whitelist
rm -f /etc/lvm/devices/system.devices 2>/dev/null || true

pvmove "${DISK_SRC}2" /dev/md11
vgreduce fedora00 "${DISK_SRC}2"
pvremove -ff -y "${DISK_SRC}2"

# 7. Update /etc/fstab with New /boot UUID
NEW_BOOT_UUID=$(blkid -s UUID -o value /dev/md10)
sed -i -E "s|^UUID=[^[:space:]]+[[:space:]]+/boot|UUID=$NEW_BOOT_UUID /boot|" /etc/fstab
umount /boot
mount /boot

# 8. Configure mdadm.conf &amp; Kernel Parameters (HP Gen8 Blueprint)
MD_BOOT_UUID=$(mdadm --detail /dev/md10 | grep -i "UUID :" | awk '{print $3}')
MD_OS_UUID=$(mdadm --detail /dev/md11 | grep -i "UUID :" | awk '{print $3}')

cat &lt;&lt; EOF > /etc/mdadm.conf
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md10 level=raid1 num-devices=2 UUID=$MD_BOOT_UUID
ARRAY /dev/md11 level=raid1 num-devices=2 UUID=$MD_OS_UUID
EOF

sed -i "s|^GRUB_CMDLINE_LINUX=.*|GRUB_CMDLINE_LINUX=\"rd.md.uuid=$MD_OS_UUID rd.lvm.lv=fedora00/root rd.md.uuid=$MD_BOOT_UUID rhgb quiet\"|" /etc/default/grub

mkdir -p /etc/dracut.conf.d
echo 'add_dracutmodules+=" mdraid lvm "' > /etc/dracut.conf.d/raid.conf
dracut -f --regenerate-all --add "mdraid lvm"
grub2-mkconfig -o /boot/grub2/grub.cfg

# 9. Partition Source Disk &amp; Complete Mirror
echo "[+] Adding source disk ($DISK_SRC) to complete mirror..."
wipefs -af "$DISK_SRC"
dd if=/dev/zero of="$DISK_SRC" bs=1M count=20 conv=fsync status=none
parted -s "$DISK_SRC" mklabel msdos
parted -s -a optimal "$DISK_SRC" mkpart primary 2048s 2099199s
parted -s -a optimal "$DISK_SRC" mkpart primary 2099200s 468861951s
parted -s "$DISK_SRC" set 1 boot on
partprobe "$DISK_SRC"
udevadm settle

mdadm --add /dev/md10 "${DISK_SRC}1"
mdadm --add /dev/md11 "${DISK_SRC}2"

# 10. Install GRUB to Both Physical MBRs
echo "[+] Installing GRUB to both MBRs..."
grub2-install --modules="part_msdos mdraid1x diskfilter ext2 lvm biosdisk" "$DISK_SRC"
grub2-install --modules="part_msdos mdraid1x diskfilter ext2 lvm biosdisk" "$DISK_DST"

echo "=========================================================="
echo "  SUCCESS: HP Gen8 RAID 1 Mirror Active &amp; Synchronized!   "
echo "  NOTE: Perform a COLD POWER CYCLE / HARD RESET to reboot."
echo "=========================================================="
cat /proc/mdstat

🎯 Key Recommendations & Takeaways

  1. Cold Hardware Power Cycle over Soft Reboot: When completing a live storage migration on enterprise server controllers (HP B120i / Intel C204), avoid systemctl reboot. Perform a full hardware reset/power cycle via iLO or power button to allow Option ROMs to negotiate clean SATA PHY links.
  2. Harden /etc/fstab with nofail: Always append nofail to secondary pools, external USB drives, and LUKS mounts so auxiliary device delays never abort the boot process into emergency mode.
  3. Handle Fedora LVM Whitelisting: Remove or update /etc/lvm/devices/system.devices when transitioning LVM from physical partitions to MD RAID devices.
  4. MBR Active Flag (0x80): Always verify set 1 boot on in parted so the Legacy BIOS executes the boot code.
  5. Declare Explicit rd.md.uuid Directives: Modern dracut initramfs requires explicit array UUIDs in GRUB_CMDLINE_LINUX to guarantee automatic array assembly on boot.
  6. Leverage iLO 4 Out-of-Band Management: Out-of-band management transforms any boot anomaly into a remote, controlled 2-minute diagnosis without requiring physical console access.

🔗 Related Guides & Series

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.