Allow specifying list of subnet IDs (rather than just a singular one)

[2rs2ts]

What would you like to be added:

I would like for an alternate option to this one in the config that allows me to specify multiple subnets to use, rather than just one. If necessary, this could be divided up into sub-lists for the private vs public subnets, although it feels like there could be a middle ground in which tags are used to distinguish the difference between those from the configuration's list.

Why is this needed:

We have two kinds of problems.

1. Non-EKS and EKS crossing wires:
   1. We have multiple sets of private subnets, but the amount is static because IPAM is hard.
   2. Because we need to support variable numbers of clusters per environment (which can shrink and grow,) we reuse the subnets.
   3. In order to support this, we intentionally do not tag the subnets with kubernetes.io/cluster/..., since there are N cluster names per environment, and we'd hit our limits on them quickly; also, defining the tags with ec2_tag is not compatible with the existing terraform definitions for the subnets.
   4. We have found that non-EKS clusters will select subnets meant for the EKS clusters, due to lexicographical ordering. We believe that the reverse would be true, but the IDs just happen to all be such that the order shakes out that way across our environments. Either way, it's troublesome.
2. We're not able to ignore an AZ we don't want to use:
   1. When we got started with AWS, we used all us-east-1 AZs available to us. However, we discovered over time that not all AZs in that DC are created equal; when nitro instances were released, one of the AZs lacked support for them for a very long time (I think it was over a year.)
   2. We decided that it would be risky to try to remove it from our network entirely, but we reorganized our then-fledgling Kubernetes terraform code to ignore the AZ that lacked nitro support. I think there were other problems besides nitro instance support, so we decided it should just forever be excluded. It's got subnets in the VPC but we don't have any nodes created in them.
   3. Nonetheless, the cloud-controller-manager selects the undesirable AZ's subnets when making LBs, causing extra unnecessary network hops.
   4. Since Introduce well-known tag for exclude subnets within a auto-discovery procedure for ELB backed services #442 was not implemented as described, we can't make the cloud-controller-manager ignore this AZ's subnets. We've considered putting in a tag with a fake cluster name to make this code not choose the subnets when auto-discovering, but that seems excessively janky and reliant upon an undocumented implementation detail that could break at any time.

Configuring subnets via annotation (as suggested here) is an unwieldy fallback. Looking up the subnet IDs in advance (or using some templating tool to do it) is clunkier than just setting it at the provider level, and mutating webhooks have proven to be unreliable and problematic for us before.

Also, we're not sure if there's some other yet-undiscovered time bomb because of the subnet auto-discovery that applies to something besides LoadBalancer syncing, so it feels like something that should just be solved at the source. Being able to specify a list of subnets, rather than just one, feels like common sense.

/kind feature

[k8s-ci-robot]

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.