Skip to content

Commit 3b535e4

Browse files
huffmancahuffmanca
authored
Merge pull request #14354 from huffmanca/OSDOCS-294
OSDOCS-294: Included CSI details.
2 parents 0e84426 + a09441b commit 3b535e4

9 files changed

+449
-0
lines changed

‎_topic_map.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -343,6 +343,8 @@ Topics:
343343
File: persistent-storage-aws
344344
- Name: Persistent Storage using iSCSI
345345
File: persistent-storage-iscsi
346+
- Name: Persistent storage using Container Storage Interface (CSI)
347+
File: persistent-storage-csi
346348
- Name: Dynamic provisioning
347349
File: dynamic-provisioning
348350
---

‎images/csi-arch.png

82.9 KB
Loading

‎modules/persistent-storage-csi-architecture.adoc

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * storage/persistent_storage/persistent-storage-csi.adoc
4+
5+
[id="persistent-storage-csi-architecture-{context}"]
6+
= CSI Architecture
7+
8+
CSI drivers are typically shipped as container images. These containers
9+
are not aware of {product-title} where they run. To use CSI-compatible
10+
storage backend in {product-title}, the cluster administrator must deploy
11+
several components that serve as a bridge between {product-title} and the
12+
storage driver.
13+
14+
The following diagram provides a high-level overview about the components
15+
running in pods in the {product-title} cluster.
16+
17+
image::../../images/csi-arch.png["Architecture of CSI components"]
18+
19+
It is possible to run multiple CSI drivers for different storage backends.
20+
Each driver needs its own external controllers' deployment and DaemonSet
21+
with the driver and CSI registrar.
22+

‎modules/persistent-storage-csi-driver-daemonset.adoc

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * storage/persistent-storage/persistent-storage-csi.adoc
4+
5+
[id="csi-driver-daemonset-{context}"]
6+
= CSI Driver DaemonSet
7+
8+
The CSI driver DaemonSet runs a pod on every node that allows
9+
{product-title} to mount storage provided by the CSI driver to the node
10+
and use it in user workloads (pods) as persistent volumes (PVs). The pod
11+
with the CSI driver installed contains the following containers:
12+
13+
* A CSI driver registrar, which registers the CSI driver into the
14+
`openshift-node` service running on the node. The `openshift-node` process
15+
running on the node then directly connects with the CSI driver using the
16+
UNIX Domain Socket available on the node.
17+
* A CSI driver.
18+
19+
The CSI driver deployed on the node should have as few credentials to the
20+
storage backend as possible. {product-title} will only use the node plug-in
21+
set of CSI calls such as `NodePublish`/`NodeUnpublish` and
22+
`NodeStage`/`NodeUnstage`, if these calls are implemented.

‎modules/persistent-storage-csi-dynamic-provisioning.adoc

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * storage/persistent-storage/persistent-storage-csi.adoc
4+
5+
[id="csi-dynamic-provisioning-{context}"]
6+
= Dynamic Provisioning
7+
8+
Dynamic provisioning of persistent storage depends on the capabilities of
9+
the CSI driver and underlying storage backend. The provider of the CSI
10+
driver should document how to create a StorageClass in {product-title} and
11+
the parameters available for configuration.
12+
13+
As seen in the OpenStack Cinder example, you can deploy this StorageClass
14+
to enable dynamic provisioning.
15+
16+
.Procedure
17+
18+
* Create a default storage class that ensures all PVCs that do not require
19+
any special storage class are provisioned by the installed CSI driver.
20+
+
21+
[source,shell]
22+
----
23+
# oc create -f - << EOF
24+
apiVersion: storage.k8s.io/v1
25+
kind: StorageClass
26+
metadata:
27+
name: cinder
28+
annotations:
29+
storageclass.kubernetes.io/is-default-class: "true"
30+
provisioner: csi-cinderplugin
31+
parameters:
32+
EOF
33+
----
34+

‎modules/persistent-storage-csi-example-deployment.adoc

Lines changed: 261 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,261 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * storage/persistent-storage/persistent-storage-csi.adoc
4+
5+
[id="csi-example-deployment-{context}"]
6+
= Example CSI deployment
7+
8+
Since {product-title} does not ship with any CSI driver installed, this
9+
example shows how to deploy a community driver for OpenStack Cinder in
10+
{product-title}.
11+
12+
.Procedure
13+
14+
. Create a new project where the CSI components will run, and then create
15+
a new service account to run the components. An explicit node selector is
16+
used to run the Daemonset with the CSI driver also on master nodes.
17+
+
18+
----
19+
# oc adm new-project csi --node-selector=""
20+
Now using project "csi" on server "https://example.com:8443".
21+
22+
# oc create serviceaccount cinder-csi
23+
serviceaccount "cinder-csi" created
24+
25+
# oc adm policy add-scc-to-user privileged system:serviceaccount:csi:cinder-csi
26+
scc "privileged" added to: ["system:serviceaccount:csi:cinder-csi"]
27+
----
28+
29+
. Apply this YAML file to create the deployment with the external CSI
30+
attacher and provisioner and DaemonSet with the CSI driver.
31+
+
32+
[source,yaml]
33+
----
34+
# This YAML file contains all API objects that are necessary to run Cinder CSI
35+
# driver.
36+
#
37+
# In production, this needs to be in separate files, e.g. service account and
38+
# role and role binding needs to be created once.
39+
#
40+
# It server as an example how to use external attacher and external provisioner
41+
# images shipped with {product-title} with a community CSI driver.
42+
43+
kind: ClusterRole
44+
apiVersion: rbac.authorization.k8s.io/v1
45+
metadata:
46+
name: cinder-csi-role
47+
rules:
48+
- apiGroups: [""]
49+
resources: ["persistentvolumes"]
50+
verbs: ["create", "delete", "get", "list", "watch", "update", "patch"]
51+
- apiGroups: [""]
52+
resources: ["events"]
53+
verbs: ["create", "get", "list", "watch", "update", "patch"]
54+
- apiGroups: [""]
55+
resources: ["persistentvolumeclaims"]
56+
verbs: ["get", "list", "watch", "update", "patch"]
57+
- apiGroups: [""]
58+
resources: ["nodes"]
59+
verbs: ["get", "list", "watch", "update", "patch"]
60+
- apiGroups: ["storage.k8s.io"]
61+
resources: ["storageclasses"]
62+
verbs: ["get", "list", "watch"]
63+
- apiGroups: ["storage.k8s.io"]
64+
resources: ["volumeattachments"]
65+
verbs: ["get", "list", "watch", "update", "patch"]
66+
- apiGroups: [""]
67+
resources: ["configmaps"]
68+
verbs: ["get", "list", "watch", "create", "update", "patch"]
69+
70+
---
71+
72+
kind: ClusterRoleBinding
73+
apiVersion: rbac.authorization.k8s.io/v1
74+
metadata:
75+
name: cinder-csi-role
76+
subjects:
77+
- kind: ServiceAccount
78+
name: cinder-csi
79+
namespace: csi
80+
roleRef:
81+
kind: ClusterRole
82+
name: cinder-csi-role
83+
apiGroup: rbac.authorization.k8s.io
84+
85+
---
86+
apiVersion: v1
87+
data:
88+
cloud.conf: W0dsb2JhbF0KYXV0aC11cmwgPSBodHRwczovL2V4YW1wbGUuY29tOjEzMDAwL3YyLjAvCnVzZXJuYW1lID0gYWxhZGRpbgpwYXNzd29yZCA9IG9wZW5zZXNhbWUKdGVuYW50LWlkID0gZTBmYTg1YjZhMDY0NDM5NTlkMmQzYjQ5NzE3NGJlZDYKcmVnaW9uID0gcmVnaW9uT25lCg== <1>
89+
kind: Secret
90+
metadata:
91+
creationTimestamp: null
92+
name: cloudconfig
93+
---
94+
kind: Deployment
95+
apiVersion: apps/v1
96+
metadata:
97+
name: cinder-csi-controller
98+
spec:
99+
replicas: 2
100+
selector:
101+
matchLabels:
102+
app: cinder-csi-controllers
103+
template:
104+
metadata:
105+
labels:
106+
app: cinder-csi-controllers
107+
spec:
108+
serviceAccount: cinder-csi
109+
containers:
110+
- name: csi-attacher
111+
image: registry.redhat.io/openshift/csi-attacher:v4.0
112+
args:
113+
- "--v=5"
114+
- "--csi-address=$(ADDRESS)"
115+
- "--leader-election"
116+
- "--leader-election-namespace=$(MY_NAMESPACE)"
117+
- "--leader-election-identity=$(MY_NAME)"
118+
env:
119+
- name: MY_NAME
120+
valueFrom:
121+
fieldRef:
122+
fieldPath: metadata.name
123+
- name: MY_NAMESPACE
124+
valueFrom:
125+
fieldRef:
126+
fieldPath: metadata.namespace
127+
- name: ADDRESS
128+
value: /csi/csi.sock
129+
volumeMounts:
130+
- name: socket-dir
131+
mountPath: /csi
132+
- name: csi-provisioner
133+
image: registry.redhat.io/openshift/csi-provisioner:v4.0
134+
args:
135+
- "--v=5"
136+
- "--provisioner=csi-cinderplugin"
137+
- "--csi-address=$(ADDRESS)"
138+
env:
139+
- name: ADDRESS
140+
value: /csi/csi.sock
141+
volumeMounts:
142+
- name: socket-dir
143+
mountPath: /csi
144+
- name: cinder-driver
145+
image: k8scloudprovider/cinder-csi-plugin:v0.3.0
146+
command: [ "/bin/cinder-csi-plugin" ]
147+
args:
148+
- "--nodeid=$(NODEID)"
149+
- "--endpoint=unix://$(ADDRESS)"
150+
- "--cloud-config=/etc/cloudconfig/cloud.conf"
151+
env:
152+
- name: NODEID
153+
valueFrom:
154+
fieldRef:
155+
fieldPath: spec.nodeName
156+
- name: ADDRESS
157+
value: /csi/csi.sock
158+
volumeMounts:
159+
- name: socket-dir
160+
mountPath: /csi
161+
- name: cloudconfig
162+
mountPath: /etc/cloudconfig
163+
volumes:
164+
- name: socket-dir
165+
emptyDir:
166+
- name: cloudconfig
167+
secret:
168+
secretName: cloudconfig
169+
170+
---
171+
172+
kind: DaemonSet
173+
apiVersion: apps/v1
174+
metadata:
175+
name: cinder-csi-ds
176+
spec:
177+
selector:
178+
matchLabels:
179+
app: cinder-csi-driver
180+
template:
181+
metadata:
182+
labels:
183+
app: cinder-csi-driver
184+
spec:
185+
<2>
186+
serviceAccount: cinder-csi
187+
containers:
188+
- name: csi-driver-registrar
189+
image: registry.redhat.io/openshift/csi-driver-registrar:v4.0
190+
securityContext:
191+
privileged: true
192+
args:
193+
- "--v=5"
194+
- "--csi-address=$(ADDRESS)"
195+
env:
196+
- name: ADDRESS
197+
value: /csi/csi.sock
198+
- name: KUBE_NODE_NAME
199+
valueFrom:
200+
fieldRef:
201+
fieldPath: spec.nodeName
202+
volumeMounts:
203+
- name: socket-dir
204+
mountPath: /csi
205+
- name: cinder-driver
206+
securityContext:
207+
privileged: true
208+
capabilities:
209+
add: ["SYS_ADMIN"]
210+
allowPrivilegeEscalation: true
211+
image: k8scloudprovider/cinder-csi-plugin:v0.3.0
212+
command: [ "/bin/cinder-csi-plugin" ]
213+
args:
214+
- "--nodeid=$(NODEID)"
215+
- "--endpoint=unix://$(ADDRESS)"
216+
- "--cloud-config=/etc/cloudconfig/cloud.conf"
217+
env:
218+
- name: NODEID
219+
valueFrom:
220+
fieldRef:
221+
fieldPath: spec.nodeName
222+
- name: ADDRESS
223+
value: /csi/csi.sock
224+
volumeMounts:
225+
- name: socket-dir
226+
mountPath: /csi
227+
- name: cloudconfig
228+
mountPath: /etc/cloudconfig
229+
- name: mountpoint-dir
230+
mountPath: /var/lib/origin/openshift.local.volumes/pods/
231+
mountPropagation: "Bidirectional"
232+
- name: cloud-metadata
233+
mountPath: /var/lib/cloud/data/
234+
- name: dev
235+
mountPath: /dev
236+
volumes:
237+
- name: cloud-metadata
238+
hostPath:
239+
path: /var/lib/cloud/data/
240+
- name: socket-dir
241+
hostPath:
242+
path: /var/lib/kubelet/plugins/csi-cinderplugin
243+
type: DirectoryOrCreate
244+
- name: mountpoint-dir
245+
hostPath:
246+
path: /var/lib/origin/openshift.local.volumes/pods/
247+
type: Directory
248+
- name: cloudconfig
249+
secret:
250+
secretName: cloudconfig
251+
- name: dev
252+
hostPath:
253+
path: /dev
254+
----
255+
<1> Replace with `cloud.conf` for your OpenStack deployment.
256+
For example, the Secret can be generated using the `oc create secret
257+
generic cloudconfig --from-file cloud.conf --dry-run -o yaml`.
258+
<2> Optionally, add `nodeSelector` to the CSI driver pod template to
259+
configure the nodes on which the CSI driver starts. Only nodes matching
260+
the selector run pods that use volumes that are served by the CSI driver.
261+
Without `nodeSelector`, the driver runs on all nodes in the cluster.

‎modules/persistent-storage-csi-external-controllers.adoc

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * storage/persistent-storage/persistent-storage-csi.adoc
4+
5+
[id="external-csi-contollers-{context}"]
6+
= External CSI controllers
7+
8+
External CSI Controllers is a deployment that deploys one or more pods
9+
with three containers:
10+
11+
* An external CSI attacher container translates `attach` and `detach`
12+
calls from {product-title} to respective `ControllerPublish` and
13+
`ControllerUnpublish` calls to the CSI driver.
14+
* An external CSI provisioner container that translates `provision` and
15+
`delete` calls from {product-title} to respective `CreateVolume` and
16+
`DeleteVolume` calls to the CSI driver.
17+
* A CSI driver container
18+
19+
The CSI attacher and CSI provisioner containers communicate with the CSI
20+
driver container using UNIX Domain Sockets, ensuring that no CSI
21+
communication leaves the pod. The CSI driver is not accessible from
22+
outside of the pod.
23+
24+
[NOTE]
25+
====
26+
`attach`, `detach`, `provision`, and `delete` operations typically require
27+
the CSI driver to use credentials to the storage backend. Run the CSI
28+
controller pods on infrastructure nodes so the credentials are never leaked
29+
to user processes, even in the event of a catastrophic security breach
30+
on a compute node.
31+
====
32+
33+
[NOTE]
34+
====
35+
The external attacher must also run for CSI drivers that do not support
36+
third-party `attach` or `detach` operations. The external attacher will
37+
not issue any `ControllerPublish` or `ControllerUnpublish` operations to
38+
the CSI driver. However, it still must run to implement the necessary
39+
{product-title} attachment API.
40+
====
41+

0 commit comments

Comments
 (0)