Skip to content

Commit b5cc46c

Browse files
abencomocaskulkarni2
abencomoc
and
askulkarni2
authored
feat: New pattern EKS AutoMode custom nodepools (#2085)
Co-authored-by: Apoorva Kulkarni <kuapoorv@amazon.com>
1 parent 6e1b7cb commit b5cc46c

12 files changed

+583
-0
lines changed

‎docs/patterns/eks-automode/eks-automode-custom-nodepools.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
title: EKS AutoMode - Custom NodePools
3+
---
4+
5+
{%
6+
include-markdown "../../../patterns/eks-automode/automode-custom-nodepools/README.md"
7+
%}

‎patterns/eks-automode/automode-custom-nodepools/README.md

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
EKS Auto Mode with custom NodePool and NodeClass
2+
---
3+
4+
This pattern deploys an EKS Auto Mode cluster with custom NodeClass and NodePool objects.
5+
6+
By default, EKS Auto Mode has two built-in NodePools to cover general compute needs. However, users often need to further customize the compute options available for different types of workloads, and segregate specific workloads onto special types of EC2 compute options (ex: amd64, arm64, GPU, etc).
7+
8+
This pattern provides an easy way to create EKS Auto Mode clusters with custom NodePools.
9+
10+
**Main features of this pattern**
11+
12+
- Creates an EKS Auto Mode cluster
13+
- Default Auto Mode NodePools are disabled
14+
- Deploys different custom NodeClass and NodePool objects, which allow fine-grained configuration of compute parameters for different use cases
15+
- amd64
16+
- arm64
17+
- gpu
18+
- Creates node IAM role and EKS Access Entry for custom NodePools
19+
- Installs default configuration to enable EBS and ELB provisioning in Auto Mode
20+
- EBS storage class
21+
- AWS LB ingress class
22+
23+
**Custom NodePool and NodeClass**
24+
25+
NodeClass and NodePool yaml manifests are provided under folder `eks-automode-config/`
26+
27+
NodeClass:
28+
29+
- `nodeclass-basic.yaml` - minimum (default) EBS configuration
30+
- `nodeclass-ebs-optimized.yaml` - optimized EBS configuration for IOPS, Size, and Throughput
31+
32+
NodePool:
33+
34+
- `nodepool-graviton.yaml` - instance families "c","r","m" using Graviton processors (arm64 architecture).
35+
- `nodepool-amd64.yaml` - instance families "c","r","m" using amd64 architecture.
36+
37+
Terraform file `eks-automode-config.tf` applies NodeClass and NodePool objects. It also creates the node IAM role and EKS Access Entry for custom nodes, and applies default EBS storage class and AWS LB ingress class.
38+
39+
To add new Node Pools and Node Classes, just add theis yaml files to the folder and update file `eks-automode-config.tf` with the added yaml file names.
40+
41+
Deploy
42+
---
43+
Apply terraform files:
44+
45+
```bash
46+
terraform apply
47+
```
48+
49+
Validate
50+
---
51+
Deploy the sample application provided in this pattern to use custom NodePools to provision nodes in the cluster.
52+
53+
```bash
54+
kubectl create ns sample-app
55+
kubectl apply -n sample-app -f sample-app.yaml
56+
```
57+
58+
This application creates StatefulSet pods using `amd64` Node Pool, and provisions EBS volumes and ALB as part of the deployment.
59+
60+
Note that Node Pools use label `NodeGroupType: amd64` and Taint `key: amd64`. The pod yaml definition includes relevant values for nodeSelector and Tolerations, to select the desired Node Pool that will provision nodes to run the pod.
61+
62+
```bash
63+
$ kubectl get nodes,pods,pvc,ingress -n sample-app
64+
NAME STATUS ROLES AGE VERSION
65+
node/i-08347a1b1ae4f01c4 Ready <none> 13m v1.31.4-eks-0f56d01
66+
67+
NAME READY STATUS RESTARTS AGE
68+
pod/httpd-0 1/1 Running 0 13m
69+
pod/httpd-1 1/1 Running 0 12m
70+
71+
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
72+
persistentvolumeclaim/httpd-logs-httpd-0 Bound pvc-8eec1429-850a-4b7c-bc78-cd399d583091 10Gi RWO auto-ebs-sc <unset> 18m
73+
persistentvolumeclaim/httpd-logs-httpd-1 Bound pvc-d86f1913-49fd-4a3f-b7a5-01da69e3ac20 10Gi RWO auto-ebs-sc <unset> 12m
74+
75+
NAME CLASS HOSTS ADDRESS PORTS AGE
76+
ingress.networking.k8s.io/httpd-ingress alb * k8s-sampleap-httpding-58bda13bc0-763023517.us-east-1.elb.amazonaws.com 80 13m
77+
```
78+
79+
Destroy
80+
---
81+
First, remove the sample app and/or any other application that you deployed ot the cluster:
82+
83+
```bash
84+
kubectl delete -n sample-app -f sample-app.yaml
85+
```
86+
87+
Then, destroy the infrastructure created with terraform:
88+
89+
```bash
90+
terraform destroy
91+
```

‎patterns/eks-automode/automode-custom-nodepools/eks-automode-config.tf

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Yaml manifests are provided under folder ./eks-automode-config/
2+
locals {
3+
storageclass_yamls = [
4+
"ebs-storageclass.yaml"
5+
]
6+
ingressclass_yamls = [
7+
"alb-ingressclass.yaml",
8+
"alb-ingressclassParams.yaml"
9+
]
10+
custom_nodeclass_yamls = [
11+
"nodeclass-basic.yaml",
12+
"nodeclass-ebs-optimized.yaml"
13+
]
14+
custom_nodepool_yamls = [
15+
"nodepool-amd64.yaml",
16+
"nodepool-graviton.yaml"
17+
]
18+
}
19+
20+
# Apply default storage class for EKS AutoMode. EBS CSI Driver runs on AWS side, managed by AWS.
21+
resource "kubectl_manifest" "storageclass_yamls" {
22+
for_each = toset(local.storageclass_yamls)
23+
24+
yaml_body = file("${path.module}/eks-automode-config/${each.value}")
25+
}
26+
27+
# Apply default ingress class for EKS AutoMode plus ingress class paramters. AWS Load Balancer Controller runs on AWS side, managed by AWS.
28+
resource "kubectl_manifest" "ingressclass_yamls" {
29+
for_each = toset(local.ingressclass_yamls)
30+
31+
yaml_body = file("${path.module}/eks-automode-config/${each.value}")
32+
}
33+
34+
# Apply custom nodeclass objects
35+
resource "kubectl_manifest" "custom_nodeclass" {
36+
for_each = toset(local.custom_nodeclass_yamls)
37+
38+
yaml_body = templatefile("${path.module}/eks-automode-config/${each.value}", {
39+
node_iam_role_name = aws_iam_role.custom_nodeclass_role.name
40+
cluster_name = module.eks.cluster_name
41+
})
42+
43+
}
44+
45+
# Apply custom nodepool objects
46+
resource "kubectl_manifest" "custom_nodepool" {
47+
for_each = toset(local.custom_nodepool_yamls)
48+
49+
yaml_body = file("${path.module}/eks-automode-config/${each.value}")
50+
51+
}

‎patterns/eks-automode/automode-custom-nodepools/eks-automode-config/alb-ingressclass.yaml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
apiVersion: networking.k8s.io/v1
2+
kind: IngressClass
3+
metadata:
4+
name: alb
5+
annotations:
6+
# Use this annotation to set an IngressClass as Default
7+
# If an Ingress doesn't specify a class, it will use the Default
8+
ingressclass.kubernetes.io/is-default-class: "true"
9+
spec:
10+
# Configures the IngressClass to use EKS Auto Mode
11+
controller: eks.amazonaws.com/alb
12+
parameters:
13+
apiGroup: eks.amazonaws.com
14+
kind: IngressClassParams
15+
name: alb

‎patterns/eks-automode/automode-custom-nodepools/eks-automode-config/alb-ingressclassParams.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
apiVersion: eks.amazonaws.com/v1
2+
kind: IngressClassParams
3+
metadata:
4+
name: alb
5+
spec:
6+
scheme: internet-facing

‎patterns/eks-automode/automode-custom-nodepools/eks-automode-config/ebs-storageclass.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
apiVersion: storage.k8s.io/v1
2+
kind: StorageClass
3+
metadata:
4+
name: auto-ebs-sc
5+
annotations:
6+
storageclass.kubernetes.io/is-default-class: "true"
7+
provisioner: ebs.csi.eks.amazonaws.com
8+
volumeBindingMode: WaitForFirstConsumer
9+
parameters:
10+
type: gp3
11+
encrypted: "true"

‎patterns/eks-automode/automode-custom-nodepools/eks-automode-config/nodeclass-basic.yaml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: eks.amazonaws.com/v1
2+
kind: NodeClass
3+
metadata:
4+
name: basic
5+
spec:
6+
# Use role name here
7+
role: "${node_iam_role_name}"
8+
subnetSelectorTerms:
9+
- tags:
10+
Name: "${cluster_name}-private*"
11+
securityGroupSelectorTerms:
12+
- tags:
13+
Name: "${cluster_name}-node"
14+
ephemeralStorage:
15+
size: "80Gi" # Range: 1-9000Gi or 1-64000G or 1-58Ti or 1-64T
16+
iops: 3000 # Range: 3000-16000
17+
throughput: 125 # Range: 125-1000
18+
tags:
19+
Environment: "production"
20+
Team: "platform"

‎patterns/eks-automode/automode-custom-nodepools/eks-automode-config/nodeclass-ebs-optimized.yaml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: eks.amazonaws.com/v1
2+
kind: NodeClass
3+
metadata:
4+
name: ebs-optimized
5+
spec:
6+
# Use role name here
7+
role: "${node_iam_role_name}"
8+
subnetSelectorTerms:
9+
- tags:
10+
Name: "${cluster_name}-private*"
11+
securityGroupSelectorTerms:
12+
- tags:
13+
Name: "${cluster_name}-node"
14+
ephemeralStorage:
15+
size: "100Gi" # Range: 1-9000Gi or 1-64000G or 1-58Ti or 1-64T
16+
iops: 5000 # Range: 3000-16000
17+
throughput: 125 # Range: 125-1000
18+
tags:
19+
Environment: "dev"
20+
Team: "data-engineering"

‎patterns/eks-automode/automode-custom-nodepools/eks-automode-config/nodepool-amd64.yaml

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
apiVersion: karpenter.sh/v1
2+
kind: NodePool
3+
metadata:
4+
name: amd64
5+
spec:
6+
template:
7+
metadata:
8+
labels:
9+
type: karpenter
10+
provisioner: amd64
11+
NodeGroupType: amd64
12+
spec:
13+
taints:
14+
- key: amd64
15+
effect: NoSchedule
16+
requirements:
17+
- key: "karpenter.sh/capacity-type"
18+
operator: In
19+
values: ["on-demand"]
20+
- key: "kubernetes.io/arch"
21+
operator: In
22+
values: ["amd64"]
23+
- key: "eks.amazonaws.com/instance-category"
24+
operator: In
25+
values: ["c", "m", "r"]
26+
- key: "eks.amazonaws.com/instance-cpu"
27+
operator: In
28+
values: ["4", "8", "16", "32"]
29+
- key: "eks.amazonaws.com/instance-hypervisor"
30+
operator: In
31+
values: ["nitro"]
32+
- key: "eks.amazonaws.com/instance-generation"
33+
operator: Gt
34+
values: ["4"]
35+
nodeClassRef:
36+
name: ebs-optimized
37+
group: eks.amazonaws.com
38+
kind: NodeClass
39+
limits:
40+
cpu: 1000

‎patterns/eks-automode/automode-custom-nodepools/eks-automode-config/nodepool-graviton.yaml

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
apiVersion: karpenter.sh/v1
2+
kind: NodePool
3+
metadata:
4+
name: graviton
5+
spec:
6+
template:
7+
metadata:
8+
labels:
9+
type: karpenter
10+
provisioner: graviton
11+
NodeGroupType: Graviton
12+
spec:
13+
taints:
14+
- key: graviton
15+
effect: NoSchedule
16+
requirements:
17+
- key: "karpenter.sh/capacity-type"
18+
operator: In
19+
values: ["on-demand"]
20+
- key: "kubernetes.io/arch"
21+
operator: In
22+
values: ["arm64"]
23+
- key: "eks.amazonaws.com/instance-category"
24+
operator: In
25+
values: ["c", "m", "r"]
26+
- key: "eks.amazonaws.com/instance-cpu"
27+
operator: In
28+
values: ["4", "8", "16", "32"]
29+
- key: "eks.amazonaws.com/instance-hypervisor"
30+
operator: In
31+
values: ["nitro"]
32+
- key: "eks.amazonaws.com/instance-generation"
33+
operator: Gt
34+
values: ["4"]
35+
nodeClassRef:
36+
name: ebs-optimized
37+
group: eks.amazonaws.com
38+
kind: NodeClass
39+
limits:
40+
cpu: 1000

0 commit comments

Comments
 (0)