Revert "Revert "Merge pull request #5 from ios-xr/dev/ebeaty_al23""

This reverts commit 6de4afa.

Author: ebeaty-cisco

README.md

@@ -13,7 +13,7 @@ You must have Packer installed, see
 [the official instructions](https://developer.hashicorp.com/packer/downloads).
 
 The template has a single mandatory argument, `kubernetes_version`, which
-is used to control the Amazon EKS optimized Amazon Linux 2 base AMI that
+is used to control the Amazon EKS optimized Amazon Linux 2023 base AMI that
 is used to generate the image and also as a tag on the produced image.
 
 To build an AMI:
@@ -47,6 +47,11 @@ packer build \
   -var 'tags={"mykey": "myvalue", "myotherkey": "myothervalue"}'
 ```
 
+Note that when building the AMI, the following warnings are expected and can be ignored:
+
+* When building the igb_uio driver, a warning for 'Skipping BTF generation' is expected. This doesn't affect the performance of the built driver.
+* When building TuneD, the 'desktop-file-install' action is expected to fail. This doesn't affect the installation of the tuned service.
+
 ## Using AMIs
 
 The AMIs produced by the template here require further configuration
@@ -70,13 +75,23 @@ Additionally, to ensure the worker node joins an EKS cluster, the normal
 EKS bootstrap script must be called.
 
 It's recommended to set this up in the User Data for the EC2 instance
-to ensure it's run on first boot. An example user data for this is:
+to ensure it's run on first boot. An example section containing this in a MIME
+multi-part user-data is:
 
 ```bash
-#!/usr/bin/env bash
+MIME-Version: 1.0
+Content-Type: multipart/mixed; boundary="BOUNDARY"
+
+<Other MIME sections, including for NodeConfig>
+
+--BOUNDARY
+Content-Type: text/x-shellscript; charset="us-ascii"
+
+#!/bin/bash
 HUGEPAGES_GB=6 ISOLATED_CORES=16-23 /etc/xrd/bootstrap.sh
-/etc/eks/bootstrap.sh my-cluster-name
 reboot
+
+--BOUNDARY--
 ```
 
 ## Image Tuning
@@ -88,6 +103,15 @@ The Packer template uses resources in this repository to run the following
 tuning steps:
   - Install the [TuneD](https://github.com/redhat-performance/tuned) tool.
   - Install an XRd TuneD profile, and run it.
+  - Set up recommended boot cmdline arguments
   - Build, install, and activate the `igb_uio` interface driver kernel module.
   - Set up recommended core handling behavior.
   - Set up hugepage handling for systems with more than one NUMA node.
+
+Note that the TuneD bootloader plugin does not work in Amazon Linux 2023. The packer template matches the cmdline arguments set by the `realtime-virtual-guest` TuneD profile, in addition to the `default_hugepagesz`, `hugepagesz` and `hugepages` arguments.
+
+## Amazon Linux 2
+
+Amazon Linux 2 is EOL.
+
+To create AMIs for AL2, see the AL2 tagged version of XRd Packer.

amazon-ebs.pkr.hcl

@@ -45,7 +45,10 @@ source "amazon-ebs" "base" {
     owners      = ["amazon"]
 
     filters = {
-      name                = format("amazon-eks-node-%s-*", var.kubernetes_version)
+      name                = format(
+        "amazon-eks-node-al2023-x86_64-standard-%s-*",
+        var.kubernetes_version
+      )
       root-device-type    = "ebs"
       virtualization-type = "hvm"
     }

files/etc/tuned/xrd-eks-node/tuned.conf renamed to files/etc/tuned/profiles/xrd-eks-node/tuned.conf

@@ -50,3 +50,11 @@ tuned-adm active
 # Enable and run the hugepage configuration service.
 systemctl start hugetlb-gigantic-pages
 systemctl enable hugetlb-gigantic-pages
+
+# Add boot cmdline args
+# The below arguments are those that the `realtime-virtual-guest` TuneD profile
+# sets, in addition to the default_hugepagesz, hugepagesz and hugepages
+# arguments. These are set here as the TuneD bootloader plugin does not work
+# in the Amazon Linux 2023 image.
+non_isolated_cores=$(/etc/xrd/get_non_isolated_cores.py $ISOLATED_CORES)
+grubby --update-kernel ALL --args "nohz_full=${ISOLATED_CORES} nohz=on skew_tick=1 intel_pstate=disable tsc=reliable nosoftlockup isolcpus=${ISOLATED_CORES} irqaffinity=${non_isolated_cores} hugepages=${boot_hugepages} rcu_nocbs=${ISOLATED_CORES} hugepagesz=1G default_hugepagesz=1G rcupdate.rcu_normal_after_boot=1"

files/etc/xrd/bootstrap.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/python3
+
+import os
+import sys
+
+def expand_range(range_str):
+    """Expand a range string like '2-4,6' into a list of integers."""
+    expanded = []
+    for part in range_str.split(','):
+        if '-' in part:
+            start, end = map(int, part.split('-'))
+            expanded.extend(range(start, end + 1))
+        else:
+            expanded.append(int(part))
+    return expanded
+
+def compress_to_range(cpu_list):
+    """Compress a list of integers into a range string like '2-4,6'."""
+    cpu_list = sorted(cpu_list)
+    ranges = []
+    start = cpu_list[0]
+    end = start
+
+    for i in range(1, len(cpu_list)):
+        if cpu_list[i] == end + 1:
+            end = cpu_list[i]
+        else:
+            if start == end:
+                ranges.append(str(start))
+            else:
+                ranges.append(f"{start}-{end}")
+            start = cpu_list[i]
+            end = start
+
+    if start == end:
+        ranges.append(str(start))
+    else:
+        ranges.append(f"{start}-{end}")
+
+    return ','.join(ranges)
+
+def exclude_cpus(exclude_set_str):
+    """Return a CPU set excluding the specified CPUs."""
+    all_cpus = set(range(os.cpu_count()))
+    exclude_cpus = set(expand_range(exclude_set_str))
+    included_cpus = sorted(all_cpus - exclude_cpus)
+    return compress_to_range(included_cpus)
+
+def main():
+    if len(sys.argv) < 2:
+        # No arguments provided, return all CPUs
+        all_cpus = list(range(os.cpu_count()))
+        print(compress_to_range(all_cpus))
+    else:
+        exclude_set_str = sys.argv[1]
+        print(exclude_cpus(exclude_set_str))
+
+if __name__ == "__main__":
+    main()

files/etc/xrd/get_non_isolated_cores.py

@@ -4,7 +4,9 @@
 set -x
 
 # Download and build the igb_uio driver, and load it into the kernel.
-yum install -y "kernel-devel-$(uname -r)"
+# N.B. A warning for 'Skipping BTF generation' is expected. This doesn't affect
+# the performance of the built driver.
+dnf install -y "kernel-devel-$(uname -r)"
 
 mkdir igb_uio
 curl https://git.dpdk.org/dpdk-kmods/snapshot/dpdk-kmods-e721c733cd24206399bebb8f0751b0387c4c1595.tar.gz | tar -xz -C igb_uio --strip-components 1
@@ -13,12 +15,10 @@ cp igb_uio/linux/igb_uio/igb_uio.ko "/lib/modules/$(uname -r)/kernel/drivers/uio
 depmod "$(uname -r)"
 rm -rf igb_uio
 
-# Download a much newer version of TuneD that available from the
-# Amazon Linux 2 repositories. This fixes several issues with the old
-# version available there.
-yum install -y \
+# TuneD is not available in the Amazon Linux 2023 repository, so download our
+# own version instead.
+dnf install -y \
     dbus \
-    dbus-python \
     ethtool \
     gawk \
     polkit \
@@ -28,16 +28,14 @@ yum install -y \
     python-linux-procfs \
     python-perf \
     python-pyudev \
-    python-schedutils \
-    tuna \
     util-linux \
     virt-what
 
 mkdir tuned
-curl -L https://github.com/redhat-performance/tuned/archive/refs/tags/v2.20.0.tar.gz | tar -xz -C tuned --strip-components 1
+curl -L https://github.com/redhat-performance/tuned/archive/refs/tags/v2.24.1.tar.gz | tar -xz -C tuned --strip-components 1
 # N.B. The 'desktop-file-install' action is expected to fail. This doesn't
 # affect the installation of the tuned service.
-make -C tuned PYTHON=/usr/bin/python2 install
+make -C tuned install
 rm -rf tuned
 
 # Set up the sysctls.