Bottlerocket node in EKS unable to join the cluster, userData not passed to kubelet that fails to load the cluster certificate

[lorenzophys]

Hello Bottlerocket community. In my team we are exploring the Bottlerocket AMIs for our EKS cluster, so we got an AMI from our platform team (we are not using a public one, but the custom ami is just a copy-paste of this variant: aws-k8s-1.29, which has id: ami-04040aebb8892ed24).

Maybe a bug report would be more suitable? #4586

Problem

The Bottlerocket (custom) AMI is unable to join the cluster: the userData passed via Karpenter's EC2NodeClass seems to be ignored and as a consequence the kubelet is unable to start:

[root@admin]# chroot /.bottlerocket/rootfs systemctl status kubelet.service
● kubelet.service - Kubelet
     Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service; enabled; preset: enabled)
    Drop-In: /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/service.d
             └─00-aws-config.conf
             /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service.d
             └─dockershim-symlink.conf
             /etc/systemd/system/kubelet.service.d
             └─exec-start.conf
             /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service.d
             └─make-kubelet-dirs.conf, prestart-load-pause-ctr.conf
     Active: activating (auto-restart) (Result: exit-code) since Mon 2025-07-14 12:47:22 UTC; 4s ago
       Docs: https://github.com/kubernetes/kubernetes
    Process: 3814 ExecStartPre=/sbin/iptables -P FORWARD ACCEPT (code=exited, status=0/SUCCESS)
    Process: 3815 ExecStartPre=/bin/ln -sf ./containerd/containerd.sock /run/dockershim.sock (code=exited, status=0/SUCCESS)
    Process: 3816 ExecStartPre=/usr/bin/mkdir -p /var/lib/kubelet/providers/secrets-store /var/lib/kubelet/node-feature-discovery/features.d (code=exited, status=0/SUCCESS)
    Process: 3817 ExecStartPre=/usr/bin/ctr --namespace=k8s.io image import --all-platforms /usr/libexec/kubernetes/kubernetes-pause.tar (code=exited, status=0/SUCCESS)
    Process: 3824 ExecStartPre=/usr/bin/ctr --namespace=k8s.io image label localhost/kubernetes/pause:0.1.0 io.cri-containerd.pinned=pinned (code=exited, status=0/SUCCESS)
    Process: 3831 ExecStart=/usr/bin/kubelet --cloud-provider external --kubeconfig /etc/kubernetes/kubelet/kubeconfig --config /etc/kubernetes/kubelet/config --container-runtime-endpoint=unix:///run/containerd/containerd.sock --containerd=/run/containerd/containerd.sock --root-dir /var/lib/kubelet --cert-dir /var/lib/kubelet/pki --image-credential-provider-bin-dir /usr/libexec/kubernetes/kubelet/plugins --image-credential-provider-config /etc/kubernetes/kubelet/credential-provider-config.yaml --hostname-override ip-xxxxxxxxxxx.eu-central-1.compute.internal --node-ip ${NODE_IP} --node-labels ${NODE_LABELS} --register-with-taints ${NODE_TAINTS} --pod-infra-container-image localhost/kubernetes/pause:0.1.0 --runtime-cgroups=/runtime.slice/containerd.service (code=exited, status=1/FAILURE)
   Main PID: 3831 (code=exited, status=1/FAILURE)
        CPU: 127ms

Jul 14 12:47:22 ip-xxxxxxxxxxxx systemd[1]: Failed to start Kubelet.

In particular:

[root@admin]# chroot /.bottlerocket/rootfs journalctl --boot
...
"Jul 14 09:19:49 localhost apiserver[1157]: 09:19:49 [INFO] Writing Settings to pending transaction in datastore: {Key { name: "settings.kubernetes.node-ip", segments: ["settings", "kubernetes", "node-ip"] }: "\"<correct-ip>\"", Key { name: "settings.kubernetes.cluster-dns-ip", segments: ["settings", "kubernetes", "cluster-dns-ip"] }: "\"172.20.0.10\"", Key { name: "settings.kubernetes.max-pods", segments: ["settings", "kubernetes", "max-pods"] }: "58", Key { name: "settings.kubernetes.provider-id", segments: ["settings", "kubernetes", "provider-id"] }: "\"aws:///eu-central-1b/i-<correct_instance_id>\"", Key { name: "settings.kubernetes.hostname-override", segments: ["settings", "kubernetes", "hostname-override"] }: "\"<correct_ip>.eu-central-1.compute.internal\""}" 
...

Which is telling me that some configuration is applied correctly, but not the one passed via userData

[root@admin]# chroot /.bottlerocket/rootfs journalctl -u kubelet
...
Jul 14 10:38:13 ip-xxxxxxxxx.aws.science.roche.com kubelet[1531]: E0714 10:38:13.720794    1531 run.go:74] "command failed" err="failed to construct kubelet dependencies: unable to load client CA file /etc/kubernetes/pki/ca.crt: error creating pool from /etc/kubernetes/pki/ca.crt: data does not contain any valid RSA or ECDSA certificates"
...

Which indicates that the cluster certificate is not recognized. In fact the /etc/kubernetes/pki/ca.crt is empty:

[root@admin]# chroot /.bottlerocket/rootfs cat /etc/kubernetes/pki/ca.crt
[root@admin]#

The kubernetes settings I get form querying the apiclient:

[root@admin]# apiclient -u /settings | jq .kubernetes
{
  "authentication-mode": "aws",
  "cloud-provider": "external",
  "cluster-dns-ip": "172.20.0.10",
  "cluster-domain": "cluster.local",
  "credential-providers": {
    "ecr-credential-provider": {
      "cache-duration": "12h",
      "enabled": true,
      "image-patterns": [
        "*.dkr.ecr.*.amazonaws.com",
        "*.dkr.ecr.*.amazonaws.com.cn",
        "*.dkr.ecr.*.on.aws",
        "*.dkr.ecr.*.on.amazonwebservices.com.cn",
        "*.dkr.ecr-fips.*.amazonaws.com",
        "*.dkr.ecr.*.cloud.adc-e.uk",
        "*.dkr.ecr-fips.*.cloud.adc-e.uk",
        "*.dkr.ecr.*.c2s.ic.gov",
        "*.dkr.ecr-fips.*.c2s.ic.gov",
        "*.dkr.ecr.*.sc2s.sgov.gov",
        "*.dkr.ecr-fips.*.sc2s.sgov.gov",
        "*.dkr.ecr.*.csp.hci.ic.gov",
        "*.dkr.ecr-fips.*.csp.hci.ic.gov",
        "public.ecr.aws"
      ]
    }
  },
  "device-ownership-from-security-context": false,
  "hostname-override": "ip-xxxxxxxxx.eu-central-1.compute.internal",
  "hostname-override-source": "private-dns-name",
  "max-pods": 58,
  "node-ip": "xxxxxxxx",
  "provider-id": "aws:///eu-central-1b/i-xxxxxxxxxxx",
  "seccomp-default": false,
  "server-tls-bootstrap": true,
  "standalone-mode": false
}

If I try to curl the cluster endpoint I get 403, which is expected since there's no certificate, but this also means that the endpoint is reachable:

[root@admin]# curl -k https://xxxxxxxxxxxxxxxxxx.gr7.eu-central-1.eks.amazonaws.com
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

Setup

We use Karpenter as autoscaler and we configured the EC2NodeClass this way:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  annotations:
  name: bottlerocketcpu
spec:
  amiFamily: Bottlerocket
  amiSelectorTerms:
    - id: ami-ourInternalAmi
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
  instanceProfile: BottlerocketKarpenterAndS3 <- this is just the standard Karpenter role on which I attached some S3 permission to easily move log files
  metadataOptions:
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpPutResponseHopLimit: 2
    httpTokens: required
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: mlops-sandbox
  subnetSelectorTerms:
    - id: subnet-ourSubnet1
    - id: subnet-ourSubnet2
  tags:
    Name: mlops-sandbox-k8s1-29-vv1-29-jun2025-Bottlerocket-Node
    Owner: RCP
    sc-product-owner: pREDi MLOps
  userData: |
    [settings]
    [settings.kubernetes]
    cluster-name = "mlops-sandbox"
    cluster-dns-ip = "172.20.0.10"
    api-server = "https://<the-cluster-endpoint>"
    cluster-certificate = "<the-cluster-certificate-base64>"
    authentication-mode = "aws"
    cloud-provider = "aws"
    log-level = "6"
  
    [settings.kubernetes.node-labels]
    "karpenter.sh/nodepool" = "bottlerocket-ondemand"

    [settings.kubernetes.node-taints]
    experimental = ["experimental:NoSchedule"]
    bottlerocket = ["node.kubernetes.io/bottlerocket:NoSchedule"]

• I take the cluster specific data from

aws eks describe-cluster --name mlops-sandbox --query 'cluster' --output text --region eu-central-1

• The instanceProfile: BottlerocketKarpenterAndS3 , contains these policies and, except for the S3 one, they're the same as all the other normally functioning nodes:

AmazonEBSCSIDriverPolicy
AmazonEC2ContainerRegistryReadOnly
AmazonEKS_CNI_Policy
AmazonEKSWorkerNodePolicy
AmazonS3FullAccess
AmazonSSMManagedInstanceCore
CloudWatchAgentServerPolicy

Version of the main components

• Kubernetes: 1.29 EKS
• Karpenter: v1.0.6
• AWS VPC CNI: v1.19.2
• Bottlerocket

[ssm-user@control]$ apiclient -u /os
{"arch":"x86_64","build_id":"807acc8b","pretty_name":"Bottlerocket OS 1.40.0 (aws-k8s-1.29)","variant_id":"aws-k8s-1.29","version_id":"1.40.0"}

Notes

Everything else in the cluster is working normally, including Karpenter and all the non-Bottlerocket nodes. I have a AWS support ticket open, so if someone from AWS is reading this and wants to take a look at the full logs and have additional info, please reach out.

I will be grateful for any help, suggestion or pointers on how to debug these type of issues.

[pyxelr]

This question has also been opened as a bug: #4586