BUG: Breaking all installs on clusters with ARM nodes.

[siimaus]

We are running cluster with mixed arm and amd nodes (managed by Karpenter). Lately we are starting to have issues:

When trying to install for example Rancher Monitoring for v2.11 charts, hook jobs from Helm charts report following:

Running /etc/scripts/..2025_07_10_12_03_05.1627751460/delete-workloads-with-old-labels.sh...
+ kubectl delete daemonset -l app=prometheus-node-exporter,release=rancher-monitoring --ignore-not-found=true
bash: line 7: /bin/kubectl: cannot execute binary file: Exec format error

This clearly indicates that at least image rancher/kuberlr-kubectl:v4.0.1 is missing arm architecture or has invalid architecture kubectl image copied to produced image.

What I was trying to do: Upgrade rancher monitoring chart on cluster, recent operation Yaml was this:

apiVersion: catalog.cattle.io/v1
kind: Operation
metadata:
  creationTimestamp: '2025-07-10T11:46:46Z'
  generation: 1
  
  name: helm-operation-n9wf8
  namespace: cattle-monitoring-system
  ownerReferences:
    - apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      name: pod-impersonation-helm-op-vd9xg
      uid: 1b8c252c-e8ab-4024-9a21-6fa0bda79b76
  resourceVersion: '702316184'
  uid: 5e80c6fe-1844-4faa-9080-908826edff65
status:
  action: upgrade
  command:
    - helm
    - upgrade
    - '--history-max=5'
    - '--install=true'
    - '--labels=catalog.cattle.io/cluster-repo-name=rancher-charts'
    - '--namespace=cattle-monitoring-system'
    - '--timeout=10m0s'
    - >-
      --values=/home/shell/helm/values-rancher-monitoring-crd-106.1.2-up69.8.2-rancher.7.yaml
    - '--version=106.1.2+up69.8.2-rancher.7'
    - '--wait=true'
    - rancher-monitoring-crd
    - /home/shell/helm/rancher-monitoring-crd-106.1.2-up69.8.2-rancher.7.tgz
    - ;
    - helm
    - upgrade
    - '--history-max=5'
    - '--install=true'
    - '--labels=catalog.cattle.io/cluster-repo-name=rancher-charts'
    - '--namespace=cattle-monitoring-system'
    - '--timeout=10m0s'
    - >-
      --values=/home/shell/helm/values-rancher-monitoring-106.1.2-up69.8.2-rancher.7.yaml
    - '--version=106.1.2+up69.8.2-rancher.7'
    - '--wait=true'
    - rancher-monitoring
    - /home/shell/helm/rancher-monitoring-106.1.2-up69.8.2-rancher.7.tgz
  conditions:
    - lastUpdateTime: '2025-07-10T11:57:45Z'
      status: 'False'
      type: Reconciling
    - lastUpdateTime: '2025-07-10T11:57:45Z'
      message: ' exit code: 123'
      reason: Stalled
      status: 'True'
      type: Stalled
  namespace: cattle-monitoring-system
  observedGeneration: 1
  podCreated: true
  podName: helm-operation-n9wf8
  podNamespace: cattle-system
  releaseName: rancher-monitoring
  token: 5mn*********************************************vw8

Log output from Helm upgrade job:

helm upgrade --history-max=5 --install=true --labels=catalog.cattle.io/cluster-repo-name=rancher-charts --namespace=cattle-monitoring-system --timeout=10m0s --values=/home/shell/helm/values-rancher-monitoring-crd-106.1.2-up69.8.2-rancher.7.yaml --version=106.1.2+up69.8.2-rancher.7 --wait=true rancher-monitoring-crd /home/shell/helm/rancher-monitoring-crd-106.1.2-up69.8.2-rancher.7.tgz
Release "rancher-monitoring-crd" has been upgraded. Happy Helming!
NAME: rancher-monitoring-crd
LAST DEPLOYED: Thu Jul 10 12:01:58 2025
NAMESPACE: cattle-monitoring-system
STATUS: deployed
REVISION: 82
TEST SUITE: None
---------------------------------------------------------------------
SUCCESS: helm upgrade --history-max=5 --install=true --labels=catalog.cattle.io/cluster-repo-name=rancher-charts --namespace=cattle-monitoring-system --timeout=10m0s --values=/home/shell/helm/values-rancher-monitoring-crd-106.1.2-up69.8.2-rancher.7.yaml --version=106.1.2+up69.8.2-rancher.7 --wait=true rancher-monitoring-crd /home/shell/helm/rancher-monitoring-crd-106.1.2-up69.8.2-rancher.7.tgz
---------------------------------------------------------------------
helm upgrade --history-max=5 --install=true --labels=catalog.cattle.io/cluster-repo-name=rancher-charts --namespace=cattle-monitoring-system --timeout=10m0s --values=/home/shell/helm/values-rancher-monitoring-106.1.2-up69.8.2-rancher.7.yaml --version=106.1.2+up69.8.2-rancher.7 --wait=true rancher-monitoring /home/shell/helm/rancher-monitoring-106.1.2-up69.8.2-rancher.7.tgz
Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
	* timed out waiting for the condition

[siimaus]

Strange this is, this runs fine on docker qemu arm64
executing on amd machine

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker run --rm -it --platform linux/arm64 --entrypoint /bin/sh rancher/kuberlr-kubectl:v4.0.1 -c 'uname -r && /bin/kubectl version'

outputs

6.8.0-1031-aws
Client Version: v1.32.0
Kustomize Version: v5.5.0

But executing on arm based EC2:

uname -r
6.1.141-155.222.amzn2023.aarch64

Run image with ctr (this is used on EKS nodes instead of docker)

# pull image
ctr -n k8s.io  image pull --platform linux/arm64 docker.io/rancher/kuberlr-kubectl:v4.0.1

# run image
ctr -n k8s.io run --rm -t docker.io/rancher/kuberlr-kubectl:v4.0.1 mytestcontainer /bin/sh

Run uname in container:

 uname -a

gives:

Linux ip-10-101-173-226.eu-central-1.compute.internal 6.1.141-155.222.amzn2023.aarch64 #1 SMP Tue Jun 17 10:29:19 UTC 2025 aarch64 GNU/Linux

Running kubectl in container:

ls -la /bin/kube* && /bin/kubectl version

gives:

lrwxrwxrwx    1 root     root             9 Feb 12 21:45 /bin/kubectl -> ./kuberlr
-rwxr-xr-x    1 root     root      50403041 Aug 30  2024 /bin/kuberlr
/bin/kubectl: line 1: can't create : Permission denied
/bin/kubectl: line 15: syntax error: unexpected ")"

Clearly, kuberlr is some kind of wrapper that unpacks stuff and then tries to execute. And it fails because it seems to be originally packed from amd binaries.

[siimaus]

Job that Rancher Monitoring Helm upgrade tries to run:

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    helm.sh/hook: pre-upgrade, pre-rollback
    helm.sh/hook-delete-policy: before-hook-creation, hook-succeeded, hook-failed
    helm.sh/hook-weight: '2'
  creationTimestamp: '2025-07-10T13:02:40Z'
  generation: 1
  labels:
    app: rancher-monitoring-upgrade
  
  name: rancher-monitoring-upgrade
  namespace: cattle-monitoring-system
  resourceVersion: '702357976'
  uid: 70ab6f5a-f676-4d0c-9109-1be8c16571df
spec:
  backoffLimit: 6
  completionMode: NonIndexed
  completions: 1
  manualSelector: false
  parallelism: 1
  podReplacementPolicy: TerminatingOrFailed
  selector:
    matchLabels:
      batch.kubernetes.io/controller-uid: 70ab6f5a-f676-4d0c-9109-1be8c16571df
  suspend: false
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: rancher-monitoring-upgrade
        batch.kubernetes.io/controller-uid: 70ab6f5a-f676-4d0c-9109-1be8c16571df
        batch.kubernetes.io/job-name: rancher-monitoring-upgrade
        controller-uid: 70ab6f5a-f676-4d0c-9109-1be8c16571df
        job-name: rancher-monitoring-upgrade
      name: rancher-monitoring-upgrade
    spec:
      containers:
        - command:
            - /bin/sh
            - '-c'
            - |
              for s in $(find /etc/scripts -type f); do 
                echo "Running $s...";
                cat $s | bash
              done;
          image: rancher/kuberlr-kubectl:v4.0.1
          imagePullPolicy: IfNotPresent
          name: run-scripts
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /etc/scripts
              name: upgrade
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext:
        runAsNonRoot: false
        runAsUser: 0
      serviceAccount: rancher-monitoring-upgrade
      serviceAccountName: rancher-monitoring-upgrade
      terminationGracePeriodSeconds: 30
      tolerations:
        - effect: NoSchedule
          key: cattle.io/os
          operator: Equal
          value: linux
      volumes:
        - configMap:
            defaultMode: 420
            name: rancher-monitoring-upgrade
          name: upgrade
status:
  failed: 6
  ready: 0
  startTime: '2025-07-10T13:02:40Z'
  terminating: 0
  uncountedTerminatedPods: {}

(One of) Pod definition(s) Rancher monitoring job tries to execute

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: '2025-07-10T13:03:54Z'
  generateName: rancher-monitoring-upgrade-
  labels:
    app: rancher-monitoring-upgrade
    batch.kubernetes.io/controller-uid: 70ab6f5a-f676-4d0c-9109-1be8c16571df
    batch.kubernetes.io/job-name: rancher-monitoring-upgrade
    controller-uid: 70ab6f5a-f676-4d0c-9109-1be8c16571df
    job-name: rancher-monitoring-upgrade  
  name: rancher-monitoring-upgrade-6c76d
  namespace: cattle-monitoring-system
  ownerReferences:
    - apiVersion: batch/v1
      blockOwnerDeletion: true
      controller: true
      kind: Job
      name: rancher-monitoring-upgrade
      uid: 70ab6f5a-f676-4d0c-9109-1be8c16571df
  resourceVersion: '702355777'
  uid: 5cd23f52-1884-450f-b259-92bc76fe47fb
spec:
  containers:
    - command:
        - /bin/sh
        - '-c'
        - |
          for s in $(find /etc/scripts -type f); do 
            echo "Running $s...";
            cat $s | bash
          done;
      image: rancher/kuberlr-kubectl:v4.0.1
      imagePullPolicy: IfNotPresent
      name: run-scripts
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /etc/scripts
          name: upgrade
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-2zwtg
          readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: ip-10-101-171-249.eu-central-1.compute.internal
  nodeSelector:
    kubernetes.io/os: linux
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext:
    runAsNonRoot: false
    runAsUser: 0
  serviceAccount: rancher-monitoring-upgrade
  serviceAccountName: rancher-monitoring-upgrade
  terminationGracePeriodSeconds: 30
  tolerations:
    - effect: NoSchedule
      key: cattle.io/os
      operator: Equal
      value: linux
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
  volumes:
    - configMap:
        defaultMode: 420
        name: rancher-monitoring-upgrade
      name: upgrade
    - name: kube-api-access-2zwtg
      projected:
        defaultMode: 420
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              items:
                - key: ca.crt
                  path: ca.crt
              name: kube-root-ca.crt
          - downwardAPI:
              items:
                - fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
                  path: namespace
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: '2025-07-10T13:03:56Z'
      status: 'False'
      type: PodReadyToStartContainers
    - lastProbeTime: null
      lastTransitionTime: '2025-07-10T13:03:54Z'
      status: 'True'
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: '2025-07-10T13:03:54Z'
      reason: PodFailed
      status: 'False'
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: '2025-07-10T13:03:54Z'
      reason: PodFailed
      status: 'False'
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: '2025-07-10T13:03:54Z'
      status: 'True'
      type: PodScheduled
  containerStatuses:
    - containerID: >-
        containerd://4dcbb81333ef7ffd5c21c796b64f6c79246cf31ab709f58149e4bdc26fff7b51
      image: docker.io/rancher/kuberlr-kubectl:v4.0.1
      imageID: >-
        docker.io/rancher/kuberlr-kubectl@sha256:31f5a30f817bd292a2572b821e7f53f4b2789248436578c57b2de17af24f71ef
      lastState: {}
      name: run-scripts
      ready: false
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: >-
            containerd://4dcbb81333ef7ffd5c21c796b64f6c79246cf31ab709f58149e4bdc26fff7b51
          exitCode: 126
          finishedAt: '2025-07-10T13:03:54Z'
          reason: Error
          startedAt: '2025-07-10T13:03:54Z'
  hostIP: 10.101.171.249
  hostIPs:
    - ip: 10.101.171.249
  phase: Failed
  podIP: 10.101.171.209
  podIPs:
    - ip: 10.101.171.209
  qosClass: BestEffort
  startTime: '2025-07-10T13:03:54Z'

[siimaus]

There is no option to use specific nodeSelector or affinity for Rancher Monitoring charts, upgrade job there have no configurable options. Replacing rancher/kuberlr-kubectl with rancher/kubectl in chart values override also does not work because this image only contains kubectl executable and no shell.
This basically breaks lot of clustrers with ARM nodes.

[siimaus]

After some readthrough, seems like it is fixed in v4.0.2 as commented in #91 (comment)
Those who have somewhere old tag lying around in deployed charts values might still have issues.