📘 Teleport Kube Agent Setup on AWS with Multi-Account, Discovery, and IAM

[pnrao1983]

This guide documents the full journey of deploying the Teleport Kube Agent using Helm with AWS IAM joining and auto-discovery features enabled. It includes real errors encountered, resolution steps, and how to verify AWS IAM setup.

🛠️ Setup Overview

Helm Values Used (aws-agent-values.yaml)

kubeClusterName: "example.teleport.com"
roles: "node, db, kube, discovery"
proxyAddr: example.teleport.com:443
joinParams:
  method: iam
  tokenName: kube-iam-token
annotations:
  serviceAccount:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/teleport-kube-role
log:
  level: "INFO"
  output: "stderr"
  format: "text"
  extraFields: ["timestamp", "level", "component", "caller"]
teleportConfig:
  discovery_service:
    enabled: true
    discovery_group: "aws-prod"
    aws:
    - types: ["eks"]
      regions: ["us-west-1"]
      tags:
        instance_metadata_tagging_req: "teleport_user@example.com"
    - types: ["eks"]
      regions: ["ap-south-2"]
      tags:
        instance_metadata_tagging_req: "teleport_user@example.com"
      assume_role_arn: "arn:aws:iam::123456789:role/teleport-kube-role"
    - types: ["ec2"]
      regions: ["us-west-1", "ap-south-2"]
      install:
        join_params:
          token_name: kube-iam-token
          method: iam
      ssm:
        document_name: "Nara-Kluster-TeleportDiscoveryInstaller"
      tags:
        env: autoenroll
  kubernetes_service:
    enabled: true
    resources:
    - labels:
        instance_metadata_tagging_req: "teleport_user@example.com"
awsDatabases:
  - types: ["rds"]
    regions: ["us-west-1"]
    tags:
      instance_metadata_tagging_req: "teleport_user@example.com"
    assume_role_arn: "arn:aws:iam::123456789:role/teleport-kube-role"
  - types: ["rds"]
    regions: ["ap-south-2"]
    tags:
      instance_metadata_tagging_req: "teleport_user@example.com"
    assume_role_arn: "arn:aws:iam::123456789:role/teleport-kube-role"

Token (Required for teleport Join)

kind: token
metadata:
  name: kube-iam-token
  revision: 137432f3-adcb-4f62-b905-544b9fe6a921
spec:
  allow:
  - aws_account: "123456789"
    aws_arn: arn:aws:sts::123456789:assumed-role/teleport-kube-role/*
  - aws_account: "123456789"
  join_method: iam
  roles:
  - Db
  - Discovery
  - Kube
  - Node
version: v2

📂 Sample Discovered Resources

Kubernetes Clusters

tsh kube ls
Kube Cluster Name        Labels                                                                                       Selected
------------------------ -------------------------------------------------------------------------------------------- --------
nara-asia-pacific-k8s    account-id=278576220453,instance_metadata_tagging_req=teleport_user@example.com,region=ap-south-2
nara-cluster             account-id=278576220453,instance_metadata_tagging_req=teleport_user@example.com,region=us-west-1
example.teleport.com

Databases

tsh db ls
Name       Description                Allowed Users Labels                                                                                                        Connect
---------- -------------------------- ------------- ------------------------------------------------------------------------------------------------------------- -------
mysql      RDS instance in us-west-1  [alice test]  account-id=278576220453,endpoint-type=instance,engine-version=8.0.40,engine=mysql,env=prod,instance_metada...
nara-mysql RDS instance in ap-south-2 [alice test]  account-id=278576220453,endpoint-type=instance,engine-version=8.0.41,engine=mysql,instance_metadata_taggin...
postgres   RDS instance in us-west-1  [alice test]  account-id=278576220453,endpoint-type=instance,engine-version=16.3,engine=postgres,instance_metadata_taggi...

SSH Nodes

tsh ls
Node Name                                  Address    Labels
------------------------------------------ ---------- ---------------------------------------------------------------------------------------------------------
ip-10-0-10-198.ap-south-2.compute.internal ⟵ Tunnel   aws/Name=nara-south-autoenroll,aws/env=autoenroll,aws/instance_metadata_tagging_req=teleport_user@example.com
ip-10-0-100-142.us-west-1.compute.internal ⟵ Tunnel   aws/Name=nara-west-1-autoenroll,aws/env=autoenroll,aws/instance_metadata_tagging_req=teleport_user@example.com

🧺s Helm Commands & Errors

First Attempt – Chart Not Found

helm upgrade teleport teleport/teleport-kluster -n teleport -f aws-values.yaml
Error: chart "teleport-kluster" matching not found in teleport index.

✅ Fixed by using the correct chart name:

helm search repo teleport
Found teleport/teleport-cluster

Release Not Found
helm upgrade teleport teleport/teleport-cluster -n teleport -f aws-values.yaml

Error: UPGRADE FAILED: "teleport" has no deployed releases

✅ Fixed by using --install:

helm upgrade --install teleport teleport/teleport-cluster -n teleport -f aws-values.yaml

❌ ServiceAccount Conflict

Error: rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount "nara-k8s-role" in namespace "teleport" exists and cannot be imported into the current release:
invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "teleport": current value is "nara-kluster"

✅ Fix: Either delete the existing ServiceAccount or reuse the same Helm release name (nara-kluster).

✅ Successful Agent Install

helm upgrade teleport-kube-agent teleport/teleport-kube-agent -n teleport-kube-agent -f aws-agent-values.yaml
Status: deployed

Validate:

kubectl describe cm teleport-kube-agent -n teleport-kube-agent
Confirmed config map includes all db_service, ssh_service, discovery_service, and kube_service settings

🔐 AWS IAM Role Setup

IAM Role: teleport-kube-role

Trust Relationship

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789:oidc-provider/oidc.eks.us-west-1.amazonaws.com/id/XXXXXXXX"
      },
      "Action": "sts:AssumeRoleWithWebIdentity"
    }
  ]
}

IAM Policy Permissions Example

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "rds:DescribeDBInstances",
        "rds:DescribeDBClusters",
        "ec2:DescribeInstances",
        "eks:DescribeCluster",
        "ssm:SendCommand",
        "ssm:GetCommandInvocation",
        "iam:PassRole"
      ],
      "Resource": "*"
    }
  ]
}

📟 Sample IAM-Related Logs

❌ Role Assumption Failure (in agent logs)

time="2025-06-05T12:34:56Z" level=error msg="Failed to assume IAM role: AccessDenied: User is not authorized to perform: sts:AssumeRole on resource arn:aws:iam::123456789:role/teleport-kube-role"

❌ EC2 AutoEnroll Error (SSM missing permission)

time="2025-06-05T12:35:20Z" level=error msg="SSM document execution failed: AccessDeniedException: User: arn:aws:sts::123456789:assumed-role/teleport-kube-role/i-0abcde123456 is not authorized to perform: ssm:SendCommand"

❌ OIDC Role Binding Issue

Error: WebIdentityErr: failed to retrieve credentials caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity

🧩 Tips for Troubleshooting IAM & Assumed Role Permissions

1. Verify Teleport Role Assumption

aws sts assume-role --role-arn arn:aws:iam::123456789:role/teleport-kube-role --role-session-name teleport-test

2. Check OIDC Provider

aws eks describe-cluster --name <your-cluster> --query "cluster.identity.oidc.issuer"

3. Test EKS Pod Identity

If the agent pod fails to assume the role:

kubectl exec -it <teleport-agent-pod> -- curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

🧹 Summary

This deployment configures a Teleport Kube Agent that:

Joins using IAM with no tokens required

Discovers EKS, EC2, and RDS resources across regions

Uses a shared IAM role and labels for multi-account resource targeting

✅ Ensure your IAM trust and policies are validated. Any mismatch between assume_role_arn, OIDC, or permissions can silently fail discovery.