Skip to content

Commit 0c9e508

Browse files
komer3komer3
authored
[Feat] Multi region support (Topology Aware Provisioning) (#280)
 --------- Co-authored-by: Khaja Omer <komer@akamai.com>
1 parent 84fe6cb commit 0c9e508

13 files changed

+471
-73
lines changed

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
- [Creating a PersistentVolumeClaim](docs/usage.md#creating-a-persistentvolumeclaim)
1919
- [Encrypted Drives using LUKS](docs/encrypted-drives.md)
2020
- [Adding Tags to Created Volumes](docs/volume-tags.md)
21+
- [Topology-Aware Provisioning](docs/topology-aware-provisioning.md)
2122
- [Development Setup](docs/development-setup.md)
2223
- [Prerequisites](docs/development-setup.md#-prerequisites)
2324
- [Setting Up the Local Development Environment](docs/development-setup.md#-setting-up-the-local-development-environment)

‎deploy/kubernetes/base/csi-storageclass.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,21 @@ metadata:
1616
provisioner: linodebs.csi.linode.com
1717
reclaimPolicy: Retain
1818
allowVolumeExpansion: true
19+
---
20+
kind: StorageClass
21+
apiVersion: storage.k8s.io/v1
22+
metadata:
23+
name: linode-block-storage-wait-for-consumer
24+
namespace: kube-system
25+
provisioner: linodebs.csi.linode.com
26+
reclaimPolicy: Delete
27+
volumeBindingMode: WaitForFirstConsumer
28+
---
29+
kind: StorageClass
30+
apiVersion: storage.k8s.io/v1
31+
metadata:
32+
name: linode-block-storage-wait-for-consumer-retain
33+
namespace: kube-system
34+
provisioner: linodebs.csi.linode.com
35+
reclaimPolicy: Retain
36+
volumeBindingMode: WaitForFirstConsumer

‎deploy/kubernetes/base/ss-csi-linode-controller.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ spec:
4343
- "--volume-name-prefix=pvc"
4444
- "--volume-name-uuid-length=16"
4545
- "--csi-address=$(ADDRESS)"
46+
- "--feature-gates=Topology=true"
4647
- "--v=2"
4748
env:
4849
- name: ADDRESS

‎docs/topology-aware-provisioning.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
## 🌐 Topology-Aware Provisioning
2+
3+
This CSI driver supports topology-aware provisioning, optimizing volume placement based on the physical infrastructure layout.
4+
5+
**Notes:**
6+
7+
1. **Volume Cloning**: Cloning only works within the same region, not across regions.
8+
2. **Volume Migration**: We can't move volumes across regions.
9+
3. **Remote Provisioning**: Volume provisioning is supported in remote regions (nodes or clusters outside of the region where the controller server is deployed).
10+
11+
> [!IMPORTANT]
12+
> Make sure you are using the latest release v0.8.6+ to utilize the remote provisioning feature.
13+
14+
#### 📝 Example StorageClass and PVC
15+
16+
```yaml
17+
allowVolumeExpansion: true
18+
apiVersion: storage.k8s.io/v1
19+
kind: StorageClass
20+
metadata:
21+
name: linode-block-storage-wait-for-consumer
22+
provisioner: linodebs.csi.linode.com
23+
reclaimPolicy: Delete
24+
volumeBindingMode: WaitForFirstConsumer
25+
---
26+
apiVersion: v1
27+
kind: PersistentVolumeClaim
28+
metadata:
29+
name: pvc-filesystem
30+
spec:
31+
accessModes:
32+
- ReadWriteOnce
33+
resources:
34+
requests:
35+
storage: 10Gi
36+
storageClassName: linode-block-storage-wait-for-consumer
37+
```
38+
39+
> **Important**: The `volumeBindingMode: WaitForFirstConsumer` setting is crucial for topology-aware provisioning. It delays volume binding and creation until a pod using the PVC is created. This allows the system to consider the pod's scheduling requirements and node assignment when selecting the most appropriate storage location, ensuring optimal data locality and performance.
40+
41+
#### 🖥️ Example Pod
42+
43+
```yaml
44+
apiVersion: v1
45+
kind: Pod
46+
metadata:
47+
name: e2e-pod
48+
spec:
49+
nodeSelector:
50+
topology.linode.com/region: us-ord
51+
tolerations:
52+
- key: "node-role.kubernetes.io/control-plane"
53+
operator: "Exists"
54+
effect: "NoSchedule"
55+
containers:
56+
- name: e2e-pod
57+
image: ubuntu
58+
command:
59+
- sleep
60+
- "1000000"
61+
volumeMounts:
62+
- mountPath: /data
63+
name: csi-volume
64+
volumes:
65+
- name: csi-volume
66+
persistentVolumeClaim:
67+
claimName: pvc-filesystem
68+
```
69+
70+
This example demonstrates how to set up topology-aware provisioning using the Linode Block Storage CSI Driver. The StorageClass defines the provisioner and reclaim policy, while the PersistentVolumeClaim requests storage from this class. The Pod specification shows how to use the PVC and includes a node selector for region-specific deployment.
71+
72+
> [!IMPORTANT]
73+
> To enable topology-aware provisioning, make sure to pass the following argument to the csi-provisioner sidecar:
74+
> ```
75+
> --feature-gates=CSINodeInfo=true
76+
> ```
77+
> This enables the CSINodeInfo feature gate, which is required for topology-aware provisioning to function correctly.
78+
>
79+
> Note: This feature is enabled by default in release v0.8.6 and later versions.
80+
81+
#### Provisioning Process
82+
83+
1. CO (Kubernetes) determines required topology based on application needs (pod scheduled region) and cluster layout.
84+
2. external-provisioner gathers topology requirements from CO and includes `TopologyRequirement` in `CreateVolume` call.
85+
3. CSI driver creates volume satisfying topology requirements.
86+
4. Driver returns actual topology of created volume.
87+
88+
By leveraging topology-aware provisioning, CSI drivers ensure optimal volume placement within the infrastructure, improving performance, availability, and data locality.

‎helm-chart/csi-driver/templates/csi-linode-controller.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ spec:
2323
- --volume-name-prefix=pvc
2424
- --volume-name-uuid-length=16
2525
- --csi-address=$(ADDRESS)
26+
- --feature-gates=Topology=true
2627
- --v=2
2728
{{- if .Values.enable_metrics}}
2829
- --metrics-address={{ .Values.csiProvisioner.metrics.address }}

‎helm-chart/csi-driver/templates/linode-block-storage-topology-aware-retain.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
apiVersion: storage.k8s.io/v1
2+
kind: StorageClass
3+
metadata:
4+
name: linode-block-storage-wait-for-consumer-retain
5+
namespace: {{ required ".Values.namespace required" .Values.namespace }}
6+
{{- if eq .Values.defaultStorageClass "linode-block-storage-wait-for-consumer-retain" }}
7+
annotations:
8+
storageclass.kubernetes.io/is-default-class: "true"
9+
{{- end }}
10+
{{- if .Values.volumeTags }}
11+
parameters:
12+
linodebs.csi.linode.com/volumeTags: {{ join "," .Values.volumeTags }}
13+
{{- end}}
14+
allowVolumeExpansion: true
15+
provisioner: linodebs.csi.linode.com
16+
reclaimPolicy: Retain
17+
volumeBindingMode: WaitForFirstConsumer

‎helm-chart/csi-driver/templates/linode-block-storage-topology-aware.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
apiVersion: storage.k8s.io/v1
2+
kind: StorageClass
3+
metadata:
4+
name: linode-block-storage-wait-for-consumer
5+
namespace: {{ required ".Values.namespace required" .Values.namespace }}
6+
{{- if eq .Values.defaultStorageClass "linode-block-storage-wait-for-consumer" }}
7+
annotations:
8+
storageclass.kubernetes.io/is-default-class: "true"
9+
{{- end }}
10+
{{- if .Values.volumeTags }}
11+
parameters:
12+
linodebs.csi.linode.com/volumeTags: {{ join "," .Values.volumeTags }}
13+
{{- end}}
14+
allowVolumeExpansion: true
15+
provisioner: linodebs.csi.linode.com
16+
reclaimPolicy: Delete
17+
volumeBindingMode: WaitForFirstConsumer

‎internal/driver/controllerserver.go‎

Lines changed: 15 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -76,24 +76,27 @@ func (cs *ControllerServer) CreateVolume(ctx context.Context, req *csi.CreateVol
7676
return &csi.CreateVolumeResponse{}, err
7777
}
7878

79-
// Create volume context
80-
volContext := cs.createVolumeContext(ctx, req)
79+
contentSource := req.GetVolumeContentSource()
80+
accessibilityRequirements := req.GetAccessibilityRequirements()
8181

8282
// Attempt to retrieve information about a source volume if the request includes a content source.
8383
// This is important for scenarios where the volume is being cloned from an existing one.
84-
sourceVolInfo, err := cs.getContentSourceVolume(ctx, req.GetVolumeContentSource())
84+
sourceVolInfo, err := cs.getContentSourceVolume(ctx, contentSource, accessibilityRequirements)
8585
if err != nil {
8686
return &csi.CreateVolumeResponse{}, err
8787
}
8888

8989
// Create the volume
90-
vol, err := cs.createAndWaitForVolume(ctx, volName, sizeGB, req.GetParameters()[VolumeTags], sourceVolInfo)
90+
vol, err := cs.createAndWaitForVolume(ctx, volName, sizeGB, req.GetParameters()[VolumeTags], sourceVolInfo, accessibilityRequirements)
9191
if err != nil {
9292
return &csi.CreateVolumeResponse{}, err
9393
}
9494

95+
// Create volume context
96+
volContext := cs.createVolumeContext(ctx, req, vol)
97+
9598
// Prepare and return response
96-
resp := cs.prepareCreateVolumeResponse(ctx, vol, size, volContext, sourceVolInfo, req.GetVolumeContentSource())
99+
resp := cs.prepareCreateVolumeResponse(ctx, vol, size, volContext, sourceVolInfo, contentSource)
97100

98101
log.V(2).Info("CreateVolume response", "response", resp)
99102
return resp, nil
@@ -154,9 +157,15 @@ func (cs *ControllerServer) ControllerPublishVolume(ctx context.Context, req *cs
154157
return resp, err
155158
}
156159

160+
// Retrieve and validate the instance associated with the Linode ID
161+
instance, err := cs.getInstance(ctx, linodeID)
162+
if err != nil {
163+
return resp, err
164+
}
165+
157166
// Check if the volume exists and is valid.
158167
// If the volume is already attached to the specified instance, it returns its device path.
159-
devicePath, err := cs.getAndValidateVolume(ctx, volumeID, linodeID)
168+
devicePath, err := cs.getAndValidateVolume(ctx, volumeID, instance, req.GetVolumeContext())
160169
if err != nil {
161170
return resp, err
162171
}
@@ -169,12 +178,6 @@ func (cs *ControllerServer) ControllerPublishVolume(ctx context.Context, req *cs
169178
}, nil
170179
}
171180

172-
// Retrieve and validate the instance associated with the Linode ID
173-
instance, err := cs.getInstance(ctx, linodeID)
174-
if err != nil {
175-
return resp, err
176-
}
177-
178181
// Check if the instance can accommodate the volume attachment
179182
if capErr := cs.checkAttachmentCapacity(ctx, instance); capErr != nil {
180183
return resp, capErr

0 commit comments

Comments
 (0)