LoadBalancer Service Provisions in Untagged VPC Subnets

[mtulio]

What happened:

A breaking change in the subnet auto-discovery of Service load balancer has been introduced by the PR: kubernetes/kubernetes#97431.

Prior to this change, the AWS Cloud Controller Manager (CCM) respected user intention by only auto-discovering subnets explicitly tagged for the cluster (kubernetes.io/cluster/<id>:owned|shared) when provisioning load balancers.

Following the mentioned change, the controller now automatically discovers all subnets in the VPC, including ones without any cluster tag. This generates an issue for users who want to install multiple clusters in the same VPC that has many subnets. The change breaks the established behavior and can lead to unintended subnet usage.

As per the PR description, this was an intentional change:

Prior to this change, the AWS cloudprovider would auto discover subnets only with the cluster tags when provisioning NLB/CLB for service resources. We want to modify this behavior to include the subnets without any cluster tags, in addition to the ones previously matched by auto-discovery. After the changes, the auto-discovery will consider all subnets except the ones tagged explicitly for other cluster. If there are multiple subnets per AZ, the ties are broken in the following order

This breaking change has impacted specific scenarios, such as:

• Untagged subnets are now being used for LoadBalancer services, violating user intent and potentially causing network issues.
• Kubernetes distributions, such as OpenShift, that install a cluster in an existing VPC with a subset of subnets are impacted, particularly after a cluster upgrade.

What you expected to happen:

The AWS Cloud Controller Manager could only auto-discover and use subnets for Service LoadBalancer provisioning that are explicitly tagged with kubernetes.io/cluster/<id>:owned|shared. This is the expected behavior to ensure the controller respects the user's defined cluster boundaries and prevents the use of unintended subnets.

How to reproduce it (as minimally and precisely as possible):

1. Start with an existing AWS VPC that has at least two public subnets in distinct zones, or create a subnet in distinct zone from existing one (without cluster tags).
2. Tag only one of these subnets with kubernetes.io/cluster/<cluster_name>:owned.
3. Deploy an OpenShift cluster into the existing VPC, ensuring it only uses the single, tagged subnet for its installation.
4. Create a new service of type LoadBalancer using a simple application deployment. For example:

apiVersion: v1
kind: Service
metadata:
  name: my-lb-service
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: my-app

5. Observe the load balancer's creation. The CCM will provision the load balancer in one of the untagged subnets, instead of the intended, tagged subnet.

Anything else we need to know?:

This issue is particularly impactful for multi-cluster environments or when deploying a single-subnet cluster within a larger, existing VPC. The lack of respect for the kubernetes.io/cluster/<id>:owned tag breaks a core assumption of how clusters are deployed in shared VPCs.

Please see the related OpenShift bug for additional context and impact reports:

• https://issues.redhat.com/browse/OCPBUGS-17432

Other relevant discussions:

• Additional subnet configuration options for AWS ELB kubernetes#97431 (comment)
• Allow specifying list of subnet IDs (rather than just a singular one) #1201

Environment:

• Kubernetes version (use kubectl version): 1.20+ (version that kubernetes/pull/97431 was backported)
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools: OpenShift
• Others:

/kind bug

[k8s-ci-robot]

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[mtulio]

cc @elmiko @JoelSpeed @Miciah

[JoelSpeed]

Observe the load balancer's creation. The CCM will provision the load balancer in one of the untagged subnets, instead of the intended, tagged subnet.

@mtulio Is the suggestion here that the order of precedence is wrong?

If there are multiple subnets per AZ, the ties are broken in the following order

What is the finish of this thought from the docs you copied?

[mtulio]

@JoelSpeed :

Observe the load balancer's creation. The CCM will provision the load balancer in one of the untagged subnets, instead of the intended, tagged subnet.

@mtulio Is the suggestion here that the order of precedence is wrong?

No, the report is that after kubernetes/kubernetes#97431 an important feature/behavior of controller had been changed (forcing discoverySubnets to use subnets outside of cluster-scope / untagged subnets), generating user's noisy and re-engineering in kubernetes distribution to workaround of the new behavior.

Considering it was introduced 10+ releases ago and now this may become de-facto a new controller behavior, I am not sure yet what to suggest, just want to bring the discussion as I didn't find the "tag standard for kubernetes.io/cluster/id" on k8s docs.

Should we need to document "well-known" tags K8s, CCMs, or some other documentation? Should we need to start increasing the scope of the e2e test scenarios here in CCM-AWS preventing any new breaking cases like that on Service loadbalancers? Would like to hear from you all suggestions.

If there are multiple subnets per AZ, the ties are broken in the following order

What is the finish of this thought from the docs you copied?

This is from the PR kubernetes/kubernetes#97431 description:

• prefer the subnet with the correct role tag. kubernetes.io/role/elb for public and kubernetes.io/role/internal-elb for private access
• prefer the subnet with the cluster tag kubernetes.io/cluster/
• prefer the subnet that is first in lexicographic order

This PR also introduces an additional annotation [...]