Skip to content

Commit bee5bfd

Browse files
slovernslovern
committed
CNF-13831 - TELCODOCS-1871 RDMA CNI for SR-IOV
1 parent b090954 commit bee5bfd

File tree

3 files changed

+179
-0
lines changed

3 files changed

+179
-0
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1550,6 +1550,8 @@ Topics:
15501550
File: configuring-sriov-net-attach
15511551
- Name: Configuring an SR-IOV InfiniBand network attachment
15521552
File: configuring-sriov-ib-attach
1553+
- Name: Configuring an RDMA subsytem for SR-IOV
1554+
File: configuring-sriov-rdma-cni
15531555
- Name: Adding a pod to an SR-IOV network
15541556
File: add-pod
15551557
- Name: Configuring interface-level network sysctl settings and all-multicast mode for SR-IOV networks

‎modules/nw-configuring-sriov-rdma-cni.adoc

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * networking/hardware_networks/configuring-sriov-rdma-cni.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="nw-sriov-configuring-sriov-rdma-cni_{context}"]
7+
= Configuring SR-IOV RDMA CNI
8+
9+
Configure an RDMA CNI on SR-IOV.
10+
11+
[NOTE]
12+
====
13+
This procedure applies only to Mellanox devices.
14+
====
15+
16+
.Prerequisites
17+
18+
* You have installed the OpenShift CLI (`oc`).
19+
* You have access to the cluster as a user with the `cluster-admin` role.
20+
* You have installed the SR-IOV Network Operator.
21+
22+
.Procedure
23+
24+
. Create an `SriovNetworkPoolConfig` CR and save it as `sriov-nw-pool.yaml`, as shown in the following example:
25+
+
26+
.Example `SriovNetworkPoolConfig` CR
27+
[source,yaml]
28+
----
29+
apiVersion: sriovnetwork.openshift.io/v1
30+
kind: SriovNetworkPoolConfig
31+
metadata:
32+
name: worker
33+
namespace: openshift-sriov-network-operator
34+
spec:
35+
maxUnavailable: 1
36+
nodeSelector:
37+
matchLabels:
38+
node-role.kubernetes.io/worker: ""
39+
rdmaMode: exclusive <1>
40+
----
41+
<1> Set RDMA network namespace mode to `exclusive`.
42+
43+
. Create the `SriovNetworkPoolConfig` resource by running the following command:
44+
+
45+
[source,terminal]
46+
----
47+
$ oc create -f sriov-nw-pool.yaml
48+
----
49+
50+
. Create an `SriovNetworkNodePolicy` CR and save it as `sriov-node-policy.yaml`, as shown in the following example:
51+
+
52+
.Example `SriovNetworkNodePolicy` CR
53+
[source,yaml]
54+
----
55+
apiVersion: sriovnetwork.openshift.io/v1
56+
kind: SriovNetworkNodePolicy
57+
metadata:
58+
name: sriov-nic-pf1
59+
namespace: openshift-sriov-network-operator
60+
spec:
61+
deviceType: netdevice
62+
isRdma: true <1>
63+
nicSelector:
64+
pfNames: ["ens3f0np0"]
65+
nodeSelector:
66+
node-role.kubernetes.io/worker: ""
67+
numVfs: 4
68+
priority: 99
69+
resourceName: sriov_nic_pf1
70+
----
71+
<1> Activate RDMA mode.
72+
73+
. Create the `SriovNetworkNodePolicy` resource by running the following command:
74+
+
75+
[source,terminal]
76+
----
77+
$ oc create -f sriov-node-policy.yaml
78+
----
79+
80+
. Create an `SriovNetwork` CR and save it as `sriov-network.yaml`, as shown in the following example:
81+
+
82+
.Example `SriovNetwork` CR
83+
[source,yaml]
84+
----
85+
apiVersion: sriovnetwork.openshift.io/v1
86+
kind: SriovNetwork
87+
metadata:
88+
name: sriov-nic-pf1
89+
namespace: openshift-sriov-network-operator
90+
spec:
91+
networkNamespace: sriov-tests
92+
resourceName: sriov_nic_pf1
93+
ipam: |-
94+
metaPlugins: |
95+
{
96+
"type": "rdma" <1>
97+
}
98+
----
99+
<1> Create the RDMA plugin.
100+
101+
. Create the `SriovNetwork` resource by running the following command:
102+
+
103+
[source,terminal]
104+
----
105+
$ oc create -f sriov-network.yaml
106+
----
107+
108+
.Verification
109+
110+
. Create a `Pod` CR and save it as `sriov-test-pod.yaml`, as shown in the following example:
111+
+
112+
.Example runtime configuration
113+
[source,yaml]
114+
----
115+
apiVersion: v1
116+
kind: Pod
117+
metadata:
118+
name: sample-pod
119+
annotations:
120+
k8s.v1.cni.cncf.io/networks: |-
121+
[
122+
{
123+
"name": "net1",
124+
"mac": "20:04:0f:f1:88:01",
125+
"ips": ["192.168.10.1/24", "2001::1/64"]
126+
}
127+
]
128+
spec:
129+
containers:
130+
- name: sample-container
131+
image: <image>
132+
imagePullPolicy: IfNotPresent
133+
command: ["sleep", "infinity"]
134+
----
135+
136+
. Create the test pod by running the following command:
137+
+
138+
[source,terminal]
139+
----
140+
$ oc create -f sriov-test-pod.yaml
141+
----
142+
143+
. Log in to the test pod by running the following command:
144+
+
145+
[source,terminal]
146+
----
147+
$ oc rsh testpod1 -n sriov-tests
148+
----
149+
150+
. Verify that the path to the `hw-counters` directory exists by running the following command:
151+
+
152+
[source,terminal]
153+
----
154+
$ ls /sys/bus/pci/devices/${PCIDEVICE_OPENSHIFT_IO_SRIOV_NIC_PF1}/infiniband/*/ports/1/hw_counters/
155+
----
156+
+
157+
.Example output
158+
[source,terminal]
159+
----
160+
duplicate_request out_of_buffer req_cqe_flush_error resp_cqe_flush_error roce_adp_retrans roce_slow_restart_trans
161+
implied_nak_seq_err out_of_sequence req_remote_access_errors resp_local_length_error roce_adp_retrans_to rx_atomic_requests
162+
lifespan packet_seq_err req_remote_invalid_request resp_remote_access_errors roce_slow_restart rx_read_requests
163+
local_ack_timeout_err req_cqe_error resp_cqe_error rnr_nak_retry_err roce_slow_restart_cnps rx_write_requests
164+
----

‎networking/hardware_networks/configuring-sriov-rdma-cni.adoc

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
[id="configuring-sriov-rdma-cni"]
3+
= Configuring an RDMA subsytem for SR-IOV
4+
include::_attributes/common-attributes.adoc[]
5+
:context: configuring-sriov-rdma-cni
6+
7+
toc::[]
8+
9+
Remote Direct Memory Access (RDMA) allows direct memory access between two systems without involving the operating system of either system.
10+
You can configure an RDMA Container Network Interface (CNI) on Single Root I/O Virtualization (SR-IOV) to enable high-performance, low-latency communication between containers.
11+
When you combine RDMA with SR-IOV, you provide a mechanism to expose hardware counters of Mellanox Ethernet devices for use inside Data Plane Development Kit (DPDK) applications.
12+
13+
include::modules/nw-configuring-sriov-rdma-cni.adoc[leveloffset=+1]

0 commit comments

Comments
 (0)