v21 Proposed Changes

[bryantbiggs]

In addition, the minimum supported versions of the AWS provider and Terraform will be v6.0 and v1.5.7 respectively. Removed arguments in v6.0 of the provider will be remediated as well

1. Remove aws-auth sub-module

   • This was announced previously when v20 was released and marked as deprecated. There are no changes to this
     module since then, nor any planned for the future, so users can continue to use the definition on its own while
     specifying a v20.x version (i.e - it continues to live as an artifact in git history, but will no longer exist on the
     master branch going forward)
   • https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/UPGRADE-
     20.0.md#%EF%B8%8F-upcoming-changes-planned-in-v210-%EF%B8%8F

2. Correct add-ons behavior

   • Hard code bootstrap_self_managed_addons to false
   • Set default value for resolve_conflicts_on_create to NONE
   • feat: Improve addon dependency chain and decrease time to provision addons (due to retries) #3218
   • feat: Added configurable delay before addons creation #3214

3. Remove identity provider correction added in 1.30

   • terraform-aws-eks/main.tf

     Lines 664 to 669 in 5583604

     	# Maintain current behavior for <= 1.29, remove default for >= 1.30
     	# `null` will return the latest Kubernetes version from the EKS API, which at time of writing is 1.30
     	# https://github.com/kubernetes/kubernetes/pull/123561
     	# TODO - remove on next breaking change in conjunction with issuer URL change below
     	idpc_backwards_compat_version = contains(["1.21", "1.22", "1.23", "1.24", "1.25", "1.26", "1.27", "1.28", "1.29"], coalesce(var.cluster_version, "1.30"))
     	idpc_issuer_url = local.idpc_backwards_compat_version ? try(aws_eks_cluster.this[0].identity[0].oidc[0].issuer, null) : null

   • Prevent conflicts between service account and jwt issuers kubernetes/kubernetes#123561

4. Correct AMI input behavior

   • Remove platform variable - legacy artifact when there were 3 OS'es (platforms). The number of AMI types and OS'es has grown quite a bit since its introduction
   • Change ami_type to be required; it will default to AL2023_x86_64_STANDARD to match similar behavior of EKS MNG starting on 1.30

5. Correct IAM permissions

   • [Cluster] Remove arn:aws:iam::aws:policy/AmazonEKSVPCResourceController - this is for SGPP and users can pass in to their node group when needed
   • [Node] Remove arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy and arn:aws:iam::aws:policy/AmazonEKS_CNI_IPv6_Policy from node roles - this will need to come via the VPC CNI pod identity
   • [Node] Replace arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly with arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly
   • [Node] Investigate adding the the node name convention via nodeadm user data in order to use arn:aws:iam::aws:policy/AmazonEKSWorkerNodeMinimalPolicy instead of arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy to better match what EKS Auto Mode does
     • How would this affect Bottlerocket? Windows?
     • Whats the benefit - need to diff the policies. Tradeoff in terms of complexity versus permissions removed.
   • [Node groups] Set hop limit to 1 with changes above

     terraform-aws-eks/node_groups.tf

     Line 5 in 5583604

     http_put_response_hop_limit = 2

6. Correct Karpenter permissions

   • Switch to v1 policy, remove prior policy remnants

     terraform-aws-eks/modules/karpenter/variables.tf

     Line 119 in 5583604

     # TODO - make v1 permssions the default policy at next breaking change

   • Enable pod identity association by default

     terraform-aws-eks/modules/karpenter/variables.tf

     Line 157 in 5583604

     # TODO - Change default to `true` at next breaking change

   • Remove IRSA? Doesn’t seem like there are any plausible situations where cross account would be useful with Karpenter so we could probably just remove IRSA entirely
   • Update node IAM role permissions with node role changes above

7. Switch to dualstack compatible OIDC endpoint for identity provider

   • create an openid_connect_provider resource for the dualstack endpoint #3237

   • terraform-aws-eks/outputs.tf

     Line 2 in 5583604

     dualstack_oidc_issuer_url = try(replace(replace(aws_eks_cluster.this[0].identity[0].oidc[0].issuer, "https://oidc.eks.", "https://oidc-eks."), ".amazonaws.com/", ".api.aws/"), null)

8. Switch to efa-only enabled by default

   • Users can specify the number of efa interfaces and at which index, but the current all efa behavior is removed

   • terraform-aws-eks/modules/eks-managed-node-group/main.tf

     Line 54 in 5583604

     interface_type = var.enable_efa_only ? contains(concat([0], var.efa_indices), i) ? "efa" : "efa-only" : "efa"

Questions

1. Should the default version for add-ons default to latest?
   • Pro: new features are pushed out through latest versions so users can utilizes those without any issues or changes
   • Con: more noise in the changes as new versions are continuously pushed out, some users have claimed that they have been burnt by new versions (unknown how valid these claims are)
2. How to make the transition to VPC CNI pod identity association easier since that will require an IAM role which currently lives in another module.
   • Current implementation is generic over the add-ons API; you pass in the add-on name you wish to enable. This would impose a double loop or loop and compare
   • This also introduces a conflict that we are trying to remove with embedding an external module (KMS)

[githubtlvmc]

Any update when the change will take effect?

[mhulscher]

@bryantbiggs I promised I write out my thoughts here as opposed to LI.

My biggest concern is with the proposed changes for karpenter, specifically removing support for IRSA:

Remove IRSA? Doesn’t seem like there are any plausible situations where cross account would be useful with Karpenter so we could probably just remove IRSA entirely

Even though I don't use karpenter in a cross-account fashion, I would still like to continue using IRSA to grant access to karpenter's kubernetes serviceaccount. The reason being that is that I don't find pod identity associations specific enough. What I mean by that is in the case pod id assocs, you specify a single IAM principal in the IAM role's trust policy, like so:

Reusability – EKS Pod Identity uses a single IAM principal instead of the separate principals for each cluster that IAM roles for service accounts use. Your IAM administrator adds the following principal to the trust policy of any role to make it usable by EKS Pod Identities.

            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            }

As per: https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html

To me, this is not specific enough. I want to specify exactly which serviceaccount, in which namespace, of which cluster, is allowed to assume this role. With pod id assocs, any EKS cluster in the same AWS account can create a binding for this IAM role. This is good enough in more straight forward scenarios where, for example, a single team owns both the entirety of EKS and IAM. Or, if there is only ever one EKS cluster per AWS account. For me, both of these are often not the case.

I consume the karpenter sub-module like so:

module "karpenter" {
  source  = "terraform-aws-modules/eks/aws//modules/karpenter"
  version = "20.37.1"

  cluster_name = var.cluster_name

  enable_irsa           = true
  enable_pod_identity   = false
  enable_v1_permissions = true

  irsa_oidc_provider_arn = var.cluster_oidc_provider_arn

  node_iam_role_additional_policies = {
    AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  }

  tags = var.tags
}

For the same reason outlined above, I don't use pod identity associations at all. I would very much appreciate if you would consider not dropping support for IRSA 🙏🏾

[bryantbiggs]

pod identity supports session tags which gives you the fine grained access scope that you are seeking aws/karpenter-provider-aws#5195

there isn't a policy provided by the Karpenter project for session tags, but that doesn't mean its not supported. in our modules, we want to reflect the upstream policies since this removes as much ambiguity and divergence as possible.

[mhulscher]

@bryantbiggs I was not aware of the ability to specify session tags, thanks for sharing. Although I'm not quite sure what it would look like, it's good to know that there's options.

Regarding your point that you want to reflect upstream policies: from what I understand by reading the AWS docs, both IRSA and pod identity associations are supported by AWS, and neither is preferred over the other by either AWS or by the wider community. There are legitimate use cases for either solution. There are also users of these tf modules that rely on IRSA. Would you not consider keeping support for IRSA to accommodate these users?

EDIT: I hadn't fully realized this at the time of writing, but I exclusively run the karpenter pods on fargate: we have no (un)managed node pools at all. And we're not going to consider using EKS auto mode because it interferes with some of our other design choices. As of yet, Fargate pods do not support pod identity associations. So that would make things extra difficult.

[raffraffraff]

Removal of IRSA isn't a good idea IMHO. Since we switched to Pod Identity Agent I've hit several issues that I never experienced with IRSA, all of them caused by PIA's requirement for a healthy agent to be running before anything else starts.

Recently we deployed a new cluster. Two workloads (one of them the CSI provider) started fine, but no webhook injected tokens, possibly due to a race condition. The pods were silently broken until we the first workload requested a PV. Only then did we see that the CSI provider didn't have permissions.

From discussions with AWS, there is no danger of IRSA deprecation. If folks want to use PIA for their later deployments after the cluster is 'stable', let them. But for critical services that run early in cluster initialization, I think IRSA is rock solid whereas PIA is not.

[bryantbiggs]

all of them caused by PIA's requirement for a healthy agent to be running before anything else starts.

This is what before_compute is for; similar issues are encountered by the VPC CNI

terraform-aws-eks/tests/fast-addons/main.tf

Line 39 in 325c3fe

before_compute = true

Also, to be clear - we are only proposing that we no longer support native IRSA authentication within the Karpenter sub-module. You can still create clusters that support IRSA, you can still use IRSA elsewhere within your applications, you can even use IRSA on the Karpenter module by providing your own trust policy. But to simplify what is already a quite complex module (set of modules), we are proposing to drop the native IRSA integration on the Karpenter sub-module

EDIT: I hadn't fully realized this at the time of writing, but I exclusively run the karpenter pods on fargate: we have no (un)managed node pools at all. And we're not going to consider using EKS auto mode because it interferes with some of our other design choices. As of yet, Fargate pods do not support pod identity associations. So that would make things extra difficult.

This was a pattern we saw that was common and then quickly died off. The challenge is that Fargate profiles are not like EC2 instances and therefore you have one way of doing things on Fargate (monitoring, observabiity, networking, security, etc.) and a completely different way on the rest of your EC2 fleet managed by Karpenter. This led to a shift of folks moving the karpenter controllers back to EKS managed nodegroup, or they would migrate to EKS Auto Mode. With EKS Auto Mode, I don't see a future for EKS Fargate (my own opinion), and therefore and less inclined to worry about all of the nuances that come with EKS Fargate

[tanvp112]

all of them caused by PIA's requirement for a healthy agent to be running before anything else starts.

This is what before_compute is for; similar issues are encountered by the VPC CNI

terraform-aws-eks/tests/fast-addons/main.tf

Line 39 in 325c3fe
before_compute = true

Also, to be clear - we are only proposing that we no longer support native IRSA authentication within the Karpenter sub-module. You can still create clusters that support IRSA, you can still use IRSA elsewhere within your applications, you can even use IRSA on the Karpenter module by providing your own trust policy. But to simplify what is already a quite complex module (set of modules), we are proposing to drop the native IRSA integration on the Karpenter sub-module

EDIT: I hadn't fully realized this at the time of writing, but I exclusively run the karpenter pods on fargate: we have no (un)managed node pools at all. And we're not going to consider using EKS auto mode because it interferes with some of our other design choices. As of yet, Fargate pods do not support pod identity associations. So that would make things extra difficult.

This was a pattern we saw that was common and then quickly died off. The challenge is that Fargate profiles are not like EC2 instances and therefore you have one way of doing things on Fargate (monitoring, observabiity, networking, security, etc.) and a completely different way on the rest of your EC2 fleet managed by Karpenter. This led to a shift of folks moving the karpenter controllers back to EKS managed nodegroup, or they would migrate to EKS Auto Mode. With EKS Auto Mode, I don't see a future for EKS Fargate (my own opinion), and therefore and less inclined to worry about all of the nuances that come with EKS Fargate

You are right if supporting all the possibilities of how EKS can be setup is not the priority; in this case Fargate only cluster is not a priority for this module. However, in real-world a Fargate only cluster is so much easier to maintain compare to MNG/NG, less ambiguity compare to auto-mode (still new)... this is especially true when upgrading a cluster with hundreds of nodes. Anyone who did this before with MNG will know what that means.

There are other examples about this module like disabling access entry is not possible with bootstrap_cluster_creator_admin_permissions hardcoded to false.

aws-auth, IRSA are currently all supported features with no EOL tipped from AWS, also NOT all addons supports PIA even few AWS managed addon doesn't. With all due respect nothing that is said here means you are making bad choice, it's your project so you make the final call. However, from users perspective the least bias/opinionated OSS modules normally standout in long run.