Software RAID (MDRaid) installation

[mmakarov-muse]

Hi.
As I researched, Talos can not be installed to MDRaid disk automatically.
It would be perfect to be able to install Talos to software raid.
Config can looks like:

machine:
  mdraid:
    mode: raid1
    name: md0
    devices:
      - /dev/sda
      - /dev/sdb

  install:
    disk: /dev/md0
    image: ghcr.io/siderolabs/installer:v1.7.0
    bootloader: true
    wipe: false

[veshant]

This would be really useful

[pratikbin]

👍

[NazarovNikita]

👍

[th1nksnow]

👍

[Francois-YACOB]

Hi all,

I am evaluating also Talos for a deployement in my datacenter as bare metal with MDRAID or ZFS Raid for my boot drives.
I didn't find any documentation to determine if it is possible or not.

I have read that Equinix and Nokia has deployed mass K8S with Talos, how do they do ?
Do they use hardware RAID or do they not implement redundancy for booting ?

I have also looked on youtube at Talos Linux Install Fest with Justin Garrison and on hardware install (ie not virtual Proxmox Azure Vmware), i didn't see any attempt for a Raid.

Thanks for the Talos team for any guidance or pointing me to the right direction (even to know that Software Raid is not possible).

François

[0xalex88]

oh, this is interesting, with the mdadm extension it sees it

NODE   NAMESPACE   TYPE               ID      VERSION   TYPE        SIZE     DISCOVERED   LABEL              PARTITIONLABEL
       runtime     DiscoveredVolume   loop1   1         disk        463 kB   squashfs
       runtime     DiscoveredVolume   loop2   1         disk        74 MB    squashfs
       runtime     DiscoveredVolume   md127   1         disk        69 GB
       runtime     DiscoveredVolume   sda     1         disk        34 GB    gpt
       runtime     DiscoveredVolume   sda1    1         partition   34 GB    zfs          92a98c58d835dd01
       runtime     DiscoveredVolume   sdb     1         disk        34 GB    gpt
       runtime     DiscoveredVolume   sdb1    1         partition   34 GB    zfs          92a98c58d835dd01
       runtime     DiscoveredVolume   sr0     1         disk        106 MB   iso9660      TALOS_V1_9_5

you can see the md127 one

[0xalex88]

So the issue now is that's getting this error when installing:

(the error locking disk one)

[cristianrat]

did you manage to set it up?

[0xalex88]

No unfortunately that's the blocker last time I've tried.

[cristianrat]

I've decided to forego raid for now, and (if) I'll run this on metal, it'll just be 3 nodes, I can accept one down if a disk will fail.
They said (in an issue here) that software raid is coming
there's always hardware raid, though :)

[Vad1mo]

I was able to install Talos on Software Raid 1, on a Hetzner Dedicated Server.

But I am not sure if I did it right, the "Talos" way. Because I see the two disks in Talos.

The primary challenge when booting an OS from software RAID on UEFI systems is ensuring the firmware can correctly load the bootloader from an EFI System Partition (ESP). Standard UEFI firmware doesn't understand mdadm software RAID. If the ESP is part of a typical mdadm array (using metadata versions like 1.2), the firmware often can't see or use it.

1. In Linux rescue system
2. Set up Raid-1
3. Flash disk

talosctl -n <IP> get disks --insecure
NODE   NAMESPACE   TYPE   ID        VERSION   SIZE     READ ONLY   TRANSPORT   ROTATIONAL   WWID                   MODEL                        SERIAL
       runtime     Disk   loop0     2         78 MB    true                                                                                     
       runtime     Disk   nvme0n1   2         512 GB   false       nvme                     eui.002538b441b2ae03   SAMSUNG MZVL2512HCJQ-00B00   S675NF0X414314
       runtime     Disk   nvme1n1   2         512 GB   false       nvme                     eui.002538b441b2adee   SAMSUNG MZVL2512HCJQ-00B00   S675NF0X414308

[ghz-max]

Can you detail a little more about your steps? Any talos enabled extension like mdadm? /proc/mdstat is as expected?

I couldn't take a lot of time to debug but I thought I had the basic and it should have worked. Definitely missed something if you were able to make it works.

Thanks!

[rdegez]

Would love some details on the step 2 & 3 as well!

[Vad1mo]

I used AI, to generate me the final script, out of the shell history. Don't execute it Copy&Paste do it in chuks.

#!/bin/bash
set -e # Exit immediately if a command exits with a non-zero status.
# set -x # Uncomment for debugging - prints each command before execution.

# --- Configuration: PLEASE VERIFY THESE CAREFULLY! ---
DISK1_DEVICE="/dev/nvme0n1" # First physical disk
DISK2_DEVICE="/dev/nvme1n1" # Second physical disk
TALOS_VERSION_TARGET="v1.10.1" # Target Talos version
ESP_PART_SIZE="+1100M"       # Size for the physical ESPs

# --- Script Variables ---
TALOS_IMG_URL="https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION_TARGET}/metal-amd64.raw.zst"
COMP_IMG_PATH="/tmp/talos-metal.raw.zst"
DECOMP_IMG_PATH="/tmp/talos-metal.raw"

RAID_DEVICE="/dev/md0"
RAID_ESP_LOGICAL_PART="${RAID_DEVICE}p1" # ESP within the RAIDed Talos image

PHYSICAL_ESP1_PART="${DISK1_DEVICE}p1"
PHYSICAL_ESP2_PART="${DISK2_DEVICE}p1"
RAID_MEMBER1_PART="${DISK1_DEVICE}p2"
RAID_MEMBER2_PART="${DISK2_DEVICE}p2"

MNT_POINT_RAID_ESP="/mnt/talos_raid_esp"
MNT_POINT_PHYS_ESP1="/mnt/phys_esp1"
MNT_POINT_PHYS_ESP2="/mnt/phys_esp2"

echo "--- Talos on RAID-1 Installer Script ---"
echo "Disks to be used: ${DISK1_DEVICE}, ${DISK2_DEVICE}"
echo "Talos Version: ${TALOS_VERSION_TARGET}"
echo "ESP Size: ${ESP_PART_SIZE}"
read -p "CAUTION: This script will wipe data on specified disks. Continue? (yes/no): " CONFIRMATION
if [ "$CONFIRMATION" != "yes" ]; then
    echo "Aborted by user."
    exit 1
fi

echo "INFO: Installing prerequisite packages..."
apt update >/dev/null
apt install -y mdadm gdisk zstd dosfstools >/dev/null

echo "INFO: Stopping any existing RAID arrays on target disks..."
mdadm --stop /dev/md* || echo "No md devices to stop or some already stopped."

echo "INFO: Wiping disks ${DISK1_DEVICE} and ${DISK2_DEVICE}..."
wipefs -a ${DISK1_DEVICE}
wipefs -a ${DISK2_DEVICE}
sgdisk --zap-all ${DISK1_DEVICE}
sgdisk --zap-all ${DISK2_DEVICE}
partprobe ${DISK1_DEVICE} || echo "WARN: partprobe ${DISK1_DEVICE} had issues. Kernel might have old table."
partprobe ${DISK2_DEVICE} || echo "WARN: partprobe ${DISK2_DEVICE} had issues. Kernel might have old table."
echo "IMPORTANT: If previous partprobe showed errors, a REBOOT OF RESCUE SYSTEM is strongly advised before re-running script."
sleep 3

echo "INFO: Partitioning ${DISK1_DEVICE}..."
sgdisk -n 1:0:${ESP_PART_SIZE} -t 1:ef00 -c 1:"EFI_System_Partition_1" ${DISK1_DEVICE}
sgdisk -n 2:0:0                -t 2:fd00 -c 2:"Linux_RAID_Member_1"    ${DISK1_DEVICE}
sgdisk --print ${DISK1_DEVICE}

echo "INFO: Partitioning ${DISK2_DEVICE}..."
sgdisk -n 1:0:${ESP_PART_SIZE} -t 1:ef00 -c 1:"EFI_System_Partition_2" ${DISK2_DEVICE}
sgdisk -n 2:0:0                -t 2:fd00 -c 2:"Linux_RAID_Member_2"    ${DISK2_DEVICE}
sgdisk --print ${DISK2_DEVICE}

echo "INFO: Informing kernel of partition changes..."
partprobe ${DISK1_DEVICE}
partprobe ${DISK2_DEVICE}
sleep 3 # Allow udev to settle

# Verify ESP sizes before formatting
echo "INFO: Verifying ESP partition sizes..."
fdisk -l ${DISK1_DEVICE}
fdisk -l ${DISK2_DEVICE}
echo "Please visually confirm p1 on both disks is ~${ESP_PART_SIZE} before proceeding."
read -p "Are ESP partition sizes correct? (yes/no): " SIZE_CONFIRM
if [ "$SIZE_CONFIRM" != "yes" ]; then
    echo "ERROR: ESP partition sizes incorrect. Please resolve partitioning. You might need to reboot the rescue system and re-run partitioning."
    exit 1
fi

echo "INFO: Formatting physical ESPs (${PHYSICAL_ESP1_PART}, ${PHYSICAL_ESP2_PART})..."
mkfs.vfat -F32 ${PHYSICAL_ESP1_PART}
mkfs.vfat -F32 ${PHYSICAL_ESP2_PART}
sync

echo "INFO: Creating RAID-1 array ${RAID_DEVICE} from ${RAID_MEMBER1_PART}, ${RAID_MEMBER2_PART}..."
mdadm --create ${RAID_DEVICE} --level=1 --raid-devices=2 ${RAID_MEMBER1_PART} ${RAID_MEMBER2_PART}
echo "INFO: RAID array created. Initial sync will happen in background."
echo "Current RAID status (Ctrl+C to stop watching after a few updates):"
watch -n 5 cat /proc/mdstat

echo "INFO: Downloading Talos image ${TALOS_VERSION_TARGET}..."
wget --progress=bar:force:noscroll -O ${COMP_IMG_PATH} ${TALOS_IMG_URL}
echo "INFO: Decompressing Talos image..."
zstd -d ${COMP_IMG_PATH} -o ${DECOMP_IMG_PATH}

echo "INFO: Writing Talos image to ${RAID_DEVICE}..."
echo "NOTE: It is best practice to wait for RAID sync to complete. Check 'cat /proc/mdstat'."
read -p "Proceed with 'dd' command now? (yes/no): " DD_CONFIRM
if [ "$DD_CONFIRM" != "yes" ]; then
    echo "Aborted 'dd' by user."
    exit 1
fi
dd if=${DECOMP_IMG_PATH} of=${RAID_DEVICE} bs=4M status=progress
sync; sync
echo "INFO: Image write complete."

echo "---------------------------------------------------------------------"
echo "MANUAL STEP REQUIRED: Fix GPT on ${RAID_DEVICE}"
echo "---------------------------------------------------------------------"
echo "Run: partprobe ${RAID_DEVICE}"
echo "Then run 'parted ${RAID_DEVICE}' interactively."
echo "  Inside parted: Type 'print'. If it warns 'Not all space used... Fix/Ignore?', type 'Fix'."
echo "  Then type 'print' again to verify, then 'quit'."
echo "Then run 'gdisk ${RAID_DEVICE}' interactively."
echo "  Inside gdisk: If it prompts to fix issues (like secondary header), type 'Y'."
echo "  Then type 'v' (verify), then 'w' (write), then 'Y' (confirm)."
echo "Finally, verify with 'fdisk -l ${RAID_DEVICE}' (should show no errors) and check 'dmesg | tail -n 30' (for no new kernel GPT errors for ${RAID_DEVICE} after a final 'partprobe ${RAID_DEVICE}')."
read -p "Press [Enter] once you have MANUALLY fixed the GPT on ${RAID_DEVICE} and verified it."

echo "INFO: Creating mount points for ESP copy..."
mkdir -p ${MNT_POINT_RAID_ESP} ${MNT_POINT_PHYS_ESP1} ${MNT_POINT_PHYS_ESP2}

mount_checked() {
    DEVICE_TO_MOUNT=$1
    MOUNT_POINT=$2
    FS_TYPE_OPT=$3 # e.g. "-t vfat" or empty for auto
    mount ${FS_TYPE_OPT} ${DEVICE_TO_MOUNT} ${MOUNT_POINT}
    if ! mountpoint -q ${MOUNT_POINT}; then
        echo "ERROR: Failed to mount ${DEVICE_TO_MOUNT} on ${MOUNT_POINT}. Check 'dmesg'. Aborting."
        # Attempt to clean up other mounts if any
        umount ${MNT_POINT_RAID_ESP} 2>/dev/null || true
        umount ${MNT_POINT_PHYS_ESP1} 2>/dev/null || true
        umount ${MNT_POINT_PHYS_ESP2} 2>/dev/null || true
        exit 1
    fi
    echo "INFO: Successfully mounted ${DEVICE_TO_MOUNT} on ${MOUNT_POINT}."
}

echo "INFO: Mounting ESPs..."
mount_checked ${RAID_ESP_LOGICAL_PART} ${MNT_POINT_RAID_ESP} ""
mount_checked ${PHYSICAL_ESP1_PART} ${MNT_POINT_PHYS_ESP1} "-t vfat"
mount_checked ${PHYSICAL_ESP2_PART} ${MNT_POINT_PHYS_ESP2} "-t vfat"

echo "INFO: Cleaning old EFI dirs and copying EFI files to physical ESPs..."
rm -rf ${MNT_POINT_PHYS_ESP1}/EFI
rm -rf ${MNT_POINT_PHYS_ESP2}/EFI
cp -a ${MNT_POINT_RAID_ESP}/EFI ${MNT_POINT_PHYS_ESP1}/
sync
cp -a ${MNT_POINT_RAID_ESP}/EFI ${MNT_POINT_PHYS_ESP2}/
sync

echo "INFO: Verifying EFI contents on physical ESPs..."
ls -lR ${MNT_POINT_PHYS_ESP1}/EFI/
ls -lR ${MNT_POINT_PHYS_ESP2}/EFI/

echo "INFO: Unmounting ESPs..."
umount ${MNT_POINT_RAID_ESP}
umount ${MNT_POINT_PHYS_ESP1}
umount ${MNT_POINT_PHYS_ESP2}
echo "INFO: Removing mount point directories..."
rmdir ${MNT_POINT_RAID_ESP} ${MNT_POINT_PHYS_ESP1} ${MNT_POINT_PHYS_ESP2} # May fail if cp created EFI inside, ok

echo "---------------------------------------------------------------------"
echo "SCRIPT COMPLETE: Disk preparation and Talos image writing finished."
echo "NEXT STEPS:"
echo "1. Reboot the server: 'shutdown -r now'"
echo "2. Enter UEFI/BIOS setup utility (use KVM/remote console)."
echo "3. CRITICAL UEFI SETTINGS:"
echo "   - Secure Boot: MUST BE DISABLED."
echo "   - Boot Mode: Must be UEFI (CSM/Legacy Disabled)."
echo "   - Boot Order: Prioritize UEFI boot from one of your physical ESPs"
echo "     (e.g., from ${PHYSICAL_ESP1_PART}, then ${PHYSICAL_ESP2_PART} as backup)."
echo "4. Save UEFI settings and allow the server to boot."
echo "---------------------------------------------------------------------"
echo "Monitor RAID sync with: watch -n 5 cat /proc/mdstat"

[Vad1mo]

fyi:

talosctl -n $SERVER_ADDRS read /proc/mdstat --talosconfig=talosconfig --context=hz-hopper

Personalities : [raid0] [raid1] [raid10] 

The software RAID (md) driver is loaded
However, no md (software RAID) arrays are currently assembled or active.