Skip to content

Commit dd94630

Browse files
bryantbiggsbryantbiggs
committed
feat: Initial pass on upgrade guide
1 parent 7d1233a commit dd94630

File tree

4 files changed

+274
-56
lines changed

4 files changed

+274
-56
lines changed

‎README.md

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -84,10 +84,6 @@ module "eks" {
8484
control_plane_subnet_ids = ["subnet-xyzde987", "subnet-slkjf456", "subnet-qeiru789"]
8585
8686
# EKS Managed Node Group(s)
87-
eks_managed_node_group_defaults = {
88-
instance_types = ["m6i.large", "m5.large", "m5n.large", "m5zn.large"]
89-
}
90-
9187
eks_managed_node_groups = {
9288
example = {
9389
# Starting on 1.30, AL2023 is the default AMI type for EKS managed node groups

‎docs/UPGRADE-21.0.md

Lines changed: 258 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,42 +7,196 @@ If you find a bug, please open an issue with supporting configuration to reprodu
77

88
- Terraform `v1.5.7` is now minimum supported version
99
- AWS provider `v6.0.0` is now minimum supported version
10+
- TLS provider `v4.0.0` is now minimum supported version
11+
- The `aws-auth` sub-module has been removed. Users who wish to utilize its functionality can continue to do so by specifying a `v20.x` version, or `~> v20.0` version constraint in their module source.
12+
- `bootstrap_self_managed_addons` is now hardcoded to `false`. This is a legacy setting and instead users should utilize the EKS addons API, which is what this module does by default. In conjunction with this change, the `bootstrap_self_managed_addons` is now ignored by the module to aid in upgrading without disruption (otherwise it would require cluster re-creation).
13+
- When enabling `enable_efa_support` or creating placement groups within a node group, users must now specify the correct `subnet_ids`; the module no longer tries to automatically select a suitable subnet.
14+
- EKS managed node group:
15+
- IMDS now default to a hop limit of 1 (previously was 2)
16+
- `ami_type` now defaults to `AL2023_x86_64_STANDARD`
17+
- `enable_monitoring` is now set to `false` by default
18+
- `enable_efa_only` is now set to `true` by default
19+
- `use_latest_ami_release_version` is now set to `true` by default
20+
- Support for autoscaling group schedules has been removed
21+
- Self-managed node group:
22+
- IMDS now default to a hop limit of 1 (previously was 2)
23+
- `ami_type` now defaults to `AL2023_x86_64_STANDARD`
24+
- `enable_monitoring` is now set to `false` by default
25+
- `enable_efa_only` is now set to `true` by default
26+
- Support for autoscaling group schedules has been removed
27+
- Karpenter:
28+
- Native support for IAM roles for service accounts (IRSA) has been removed; EKS Pod Identity is now enabled by default
29+
- Karpenter controller policy for prior to Karpenter `v1` have been removed (i.e. `v0.33`); the `v1` policy is now used by default
30+
- `create_pod_identity_association` is now set to `true` by default
31+
- `addons.resolve_conflicts_on_create` is now set to `"NONE"` by default (was `"OVERWRITE"`).
32+
- `addons.most_recent` is now set to `true` by default (was `false`).
33+
- `cluster_identity_providers.issuer_url` is now required to be set by users; the prior incorrect default has been removed. See https://github.com/terraform-aws-modules/terraform-aws-eks/pull/3055 and https://github.com/kubernetes/kubernetes/pull/123561 for more details.
1034

1135
## Additional changes
1236

1337
### Added
1438

1539
- Support for `region` parameter to specify the AWS region for the resources created if different from the provider region.
40+
- Both the EKS managed and self-managed node groups now support creating their own security groups (again). This is primarily motivated by the changes for EFA support; previously users would need to specify `enable_efa_support` both at the cluster level (to add the appropriate security group rules to the shared node security group) as well as the node group level. However, its not always desirable to have these rules across ALL node groups when they are really only required on the node group where EFA is utilized. And similarly for other use cases, users can create custom rules for a specific node group instead of apply across ALL node groups.
1641

1742
### Modified
1843

1944
- Variable definitions now contain detailed `object` types in place of the previously used any type.
45+
- The embedded KMS key module definition has been updated to `v4.0` to support the same version requirements as well as the new `region` argument.
2046

2147
### Variable and output changes
2248

2349
1. Removed variables:
2450

25-
-
51+
- `enable_efa_support` - users only need to set this within the node group configuration, as the module no longer manages EFA support at the cluster level.
52+
- `enable_security_groups_for_pods` - users can instead attach the `arn:aws:iam::aws:policy/AmazonEKSVPCResourceController` policy via `iam_role_additional_policies` if using security groups for pods.
53+
- `eks-managed-node-group` sub-module
54+
- `cluster_service_ipv4_cidr` - users should use `cluster_service_cidr` instead (for either IPv4 or IPv6).
55+
- `elastic_gpu_specifications`
56+
- `elastic_inference_accelerator`
57+
- `platform` - this is superseded by `ami_type`
58+
- `placement_group_strategy` - set to `cluster` by the module
59+
- `placement_group_az` - users will need to specify the correct subnet in `subnet_ids`
60+
- `create_schedule`
61+
- `schedules`
62+
- `self-managed-node-group` sub-module
63+
- `elastic_gpu_specifications`
64+
- `elastic_inference_accelerator`
65+
- `platform` - this is superseded by `ami_type`
66+
- `create_schedule`
67+
- `schedules`
68+
- `placement_group_az` - users will need to specify the correct subnet in `subnet_ids`
69+
- `hibernation_options` - not valid in EKS
70+
- `min_elb_capacity` - not valid in EKS
71+
- `wait_for_elb_capacity` - not valid in EKS
72+
- `wait_for_capacity_timeout` - not valid in EKS
73+
- `default_cooldown` - not valid in EKS
74+
- `target_group_arns` - not valid in EKS
75+
- `service_linked_role_arn` - not valid in EKS
76+
- `warm_pool` - not valid in EKS
77+
- `fargate-profile` sub-module
78+
- None
79+
- `karpenter` sub-module
80+
- `enable_v1_permissions` - v1 permissions are now the default
81+
- `enable_irsa`
82+
- `irsa_oidc_provider_arn`
83+
- `irsa_namespace_service_accounts`
84+
- `irsa_assume_role_condition_test`
2685

2786
2. Renamed variables:
2887

29-
-
88+
- Variables prefixed with `cluster_*` have been stripped of the prefix to better match the underlying API:
89+
- `cluster_name` -> `name`
90+
- `cluster_version` -> `kubernetes_version`
91+
- `cluster_enabled_log_types` -> `enabled_log_types`
92+
- `cluster_force_update_version` -> `force_update_version`
93+
- `cluster_compute_config` -> `compute_config`
94+
- `cluster_upgrade_policy` -> `upgrade_policy`
95+
- `cluster_remote_network_config` -> `remote_network_config`
96+
- `cluster_zonal_shift_config` -> `zonal_shift_config`
97+
- `cluster_additional_security_group_ids` -> `additional_security_group_ids`
98+
- `cluster_endpoint_private_access` -> `endpoint_private_access`
99+
- `cluster_endpoint_public_access` -> `endpoint_public_access`
100+
- `cluster_endpoint_public_access_cidrs` -> `endpoint_public_access_cidrs`
101+
- `cluster_ip_family` -> `ip_family`
102+
- `cluster_service_ipv4_cidr` -> `service_ipv4_cidr`
103+
- `cluster_service_ipv6_cidr` -> `service_ipv6_cidr`
104+
- `cluster_encryption_config` -> `encryption_config`
105+
- `create_cluster_primary_security_group_tags` -> `create_primary_security_group_tags`
106+
- `cluster_timeouts` -> `timeouts`
107+
- `create_cluster_security_group` -> `create_security_group`
108+
- `cluster_security_group_id` -> `security_group_id`
109+
- `cluster_security_group_name` -> `security_group_name`
110+
- `cluster_security_group_use_name_prefix` -> `security_group_use_name_prefix`
111+
- `cluster_security_group_description` -> `security_group_description`
112+
- `cluster_security_group_additional_rules` -> `security_group_additional_rules`
113+
- `cluster_security_group_tags` -> `security_group_tags`
114+
- `cluster_encryption_policy_use_name_prefix` -> `encryption_policy_use_name_prefix`
115+
- `cluster_encryption_policy_name` -> `encryption_policy_name`
116+
- `cluster_encryption_policy_description` -> `encryption_policy_description`
117+
- `cluster_encryption_policy_path` -> `encryption_policy_path`
118+
- `cluster_encryption_policy_tags` -> `encryption_policy_tags`
119+
- `cluster_addons` -> `addons`
120+
- `cluster_addons_timeouts` -> `addons_timeouts`
121+
- `cluster_identity_providers` -> `identity_providers`
122+
- `eks-managed-node-group` sub-module
123+
- `cluster_version` -> `kubernetes_version`
124+
- `self-managed-node-group` sub-module
125+
- `cluster_version` -> `kubernetes_version`
126+
- `delete_timeout` -> `timeouts`
127+
- `fargate-profile` sub-module
128+
- None
129+
- `karpenter` sub-module
130+
- None
30131

31132
3. Added variables:
32133

33-
-
134+
- `region`
135+
- `eks-managed-node-group` sub-module
136+
- `region`
137+
- `partition` - added to reduce number of `GET` requests from data sources when possible
138+
- `account_id` - added to reduce number of `GET` requests from data sources when possible
139+
- `create_security_group`
140+
- `security_group_name`
141+
- `security_group_use_name_prefix`
142+
- `security_group_description`
143+
- `security_group_ingress_rules`
144+
- `security_group_egress_rules`
145+
- `security_group_tags`
146+
- `self-managed-node-group` sub-module
147+
- `region`
148+
- `partition` - added to reduce number of `GET` requests from data sources when possible
149+
- `account_id` - added to reduce number of `GET` requests from data sources when possible
150+
- `create_security_group`
151+
- `security_group_name`
152+
- `security_group_use_name_prefix`
153+
- `security_group_description`
154+
- `security_group_ingress_rules`
155+
- `security_group_egress_rules`
156+
- `security_group_tags`
157+
- `fargate-profile` sub-module
158+
- `region`
159+
- `partition` - added to reduce number of `GET` requests from data sources when possible
160+
- `account_id` - added to reduce number of `GET` requests from data sources when possible
161+
- `karpenter` sub-module
162+
- `region`
34163

35164
4. Removed outputs:
36165

37-
-
166+
- `eks-managed-node-group` sub-module
167+
- `platform` - this is superseded by `ami_type`
168+
- `autoscaling_group_schedule_arns`
169+
- `self-managed-node-group` sub-module
170+
- `platform` - this is superseded by `ami_type`
171+
- `autoscaling_group_schedule_arns`
172+
- `fargate-profile` sub-module
173+
- None
174+
- `karpenter` sub-module
175+
- None
38176

39177
5. Renamed outputs:
40178

41-
-
179+
- `eks-managed-node-group` sub-module
180+
- None
181+
- `self-managed-node-group` sub-module
182+
- None
183+
- `fargate-profile` sub-module
184+
- None
185+
- `karpenter` sub-module
186+
- None
42187

43188
6. Added outputs:
44189

45-
-
190+
- `eks-managed-node-group` sub-module
191+
- `security_group_arn`
192+
- `security_group_id`
193+
- `self-managed-node-group` sub-module
194+
- `security_group_arn`
195+
- `security_group_id`
196+
- `fargate-profile` sub-module
197+
- None
198+
- `karpenter` sub-module
199+
- None
46200

47201
## Upgrade Migrations
48202

@@ -54,7 +208,56 @@ module "eks" {
54208
version = "~> 20.0"
55209
56210
# Truncated for brevity ...
211+
# Not all renamed variables are shown here, please refer to the full list above.
57212
213+
enable_efa_support = true
214+
215+
eks_managed_node_group_defaults = {
216+
iam_role_additional_policies = {
217+
AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
218+
}
219+
}
220+
221+
eks_managed_node_groups = {
222+
efa = {
223+
ami_type = "AL2023_x86_64_NVIDIA"
224+
instance_types = ["p5e.48xlarge"]
225+
226+
enable_efa_support = true
227+
enable_efa_only = true
228+
}
229+
}
230+
231+
self_managed_node_groups = {
232+
example = {
233+
use_mixed_instances_policy = true
234+
mixed_instances_policy = {
235+
instances_distribution = {
236+
on_demand_base_capacity = 0
237+
on_demand_percentage_above_base_capacity = 0
238+
on_demand_allocation_strategy = "lowest-price"
239+
spot_allocation_strategy = "price-capacity-optimized"
240+
}
241+
242+
# ASG configuration
243+
override = [
244+
{
245+
instance_requirements = {
246+
cpu_manufacturers = ["intel"]
247+
instance_generations = ["current", "previous"]
248+
spot_max_price_percentage_over_lowest_price = 100
249+
250+
vcpu_count = {
251+
min = 1
252+
}
253+
254+
allowed_instance_types = ["t*", "m*"]
255+
}
256+
}
257+
]
258+
}
259+
}
260+
}
58261
}
59262
```
60263

@@ -66,7 +269,56 @@ module "eks" {
66269
version = "~> 21.0"
67270
68271
# Truncated for brevity ...
272+
# Not all renamed variables are shown here, please refer to the full list above.
273+
274+
eks_managed_node_groups = {
275+
efa = {
276+
ami_type = "AL2023_x86_64_NVIDIA"
277+
instance_types = ["p5e.48xlarge"]
278+
279+
iam_role_additional_policies = {
280+
AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
281+
}
282+
283+
enable_efa_support = true
284+
285+
subnet_ids = element(module.vpc.private_subnets, 0)
286+
}
287+
}
288+
289+
self_managed_node_groups = {
290+
example = {
291+
use_mixed_instances_policy = true
292+
mixed_instances_policy = {
293+
instances_distribution = {
294+
on_demand_base_capacity = 0
295+
on_demand_percentage_above_base_capacity = 0
296+
on_demand_allocation_strategy = "lowest-price"
297+
spot_allocation_strategy = "price-capacity-optimized"
298+
}
299+
300+
# ASG configuration
301+
# Need to wrap in `launch_template` now
302+
launch_template = {
303+
override = [
304+
{
305+
instance_requirements = {
306+
cpu_manufacturers = ["intel"]
307+
instance_generations = ["current", "previous"]
308+
spot_max_price_percentage_over_lowest_price = 100
309+
310+
vcpu_count = {
311+
min = 1
312+
}
69313
314+
allowed_instance_types = ["t*", "m*"]
315+
}
316+
}
317+
]
318+
}
319+
}
320+
}
321+
}
70322
}
71323
```
72324

‎docs/compute_resources.md

Lines changed: 0 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -153,41 +153,3 @@ See the [`examples/self-managed-node-group/` example](https://github.com/terrafo
153153
### Fargate Profiles
154154

155155
Fargate profiles are straightforward to use and therefore no further details are provided here. See the [`tests/fargate-profile/` tests](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/tests/fargate-profile) for a working example of various configurations.
156-
157-
### Default Configurations
158-
159-
Each type of compute resource (EKS managed node group, self managed node group, or Fargate profile) provides the option for users to specify a default configuration. These default configurations can be overridden from within the compute resource's individual definition. The order of precedence for configurations (from highest to least precedence):
160-
161-
- Compute resource individual configuration
162-
- Compute resource family default configuration (`eks_managed_node_group_defaults`, `self_managed_node_group_defaults`, `fargate_profile_defaults`)
163-
- Module default configuration (see `variables.tf` and `node_groups.tf`)
164-
165-
For example, the following creates 4 AWS EKS Managed Node Groups:
166-
167-
```hcl
168-
eks_managed_node_group_defaults = {
169-
ami_type = "AL2023_x86_64_STANDARD"
170-
disk_size = 50
171-
instance_types = ["m6i.large", "m5.large", "m5n.large", "m5zn.large"]
172-
}
173-
174-
eks_managed_node_groups = {
175-
# Uses module default configurations overridden by configuration above
176-
default = {}
177-
178-
# This further overrides the instance types used
179-
compute = {
180-
instance_types = ["c5.large", "c6i.large", "c6d.large"]
181-
}
182-
183-
# This further overrides the instance types
184-
persistent = {
185-
instance_types = ["r5.xlarge", "r6i.xlarge", "r5b.xlarge"]
186-
}
187-
188-
# This overrides the OS used
189-
bottlerocket = {
190-
ami_type = "BOTTLEROCKET_x86_64"
191-
}
192-
}
193-
```

0 commit comments

Comments
 (0)