Skip to content

Commit 15596db

Browse files
kquinn1204kquinn1204
committed
TELCODOCS-1683 QinQ support in SR-IOV Network Operator 1
1 parent fefa719 commit 15596db

File tree

4 files changed

+229
-0
lines changed

4 files changed

+229
-0
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1408,6 +1408,8 @@ Topics:
14081408
File: add-pod
14091409
- Name: Configuring interface-level network sysctl settings and all-multicast mode for SR-IOV networks
14101410
File: configuring-interface-sysctl-sriov-device
1411+
- Name: Configuring QinQ support for SR-IOV networks
1412+
File: configuring-sriov-qinq-support
14111413
- Name: Using high performance multicast
14121414
File: using-sriov-multicast
14131415
- Name: Using DPDK and RDMA

‎modules/nw-configuring-qinq-sriov-proc.adoc

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * networking/hardware_networks/configuring-sriov-qinq-support.adoc
4+
// *
5+
// *
6+
7+
:_mod-docs-content-type: PROCEDURE
8+
9+
[id="nw-configuring-qinq-sriov-proc_{context}"]
10+
= Configuring QinQ support for SR-IOV enabled workloads
11+
12+
.Prerequisites
13+
14+
* You have installed the OpenShift CLI (`oc`).
15+
* You have access to the cluster as a user with the `cluster-admin` role.
16+
* You have installed the SR-IOV Network Operator.
17+
18+
.Procedure
19+
20+
. Create a file named `sriovnetpolicy-810-sriov-node-network.yaml` by using the following content:
21+
+
22+
[source,yaml]
23+
----
24+
apiVersion: sriovnetwork.openshift.io/v1
25+
kind: SriovNetworkNodePolicy
26+
metadata:
27+
name: sriovnetpolicy-810
28+
namespace: openshift-sriov-network-operator
29+
spec:
30+
deviceType: netdevice
31+
nicSelector:
32+
pfNames:
33+
- ens5f0#0-9
34+
nodeSelector:
35+
node-role.kubernetes.io/worker-cnf: ""
36+
numVfs: 10
37+
priority: 99
38+
resourceName: resource810
39+
----
40+
41+
. Create the `SriovNetworkNodePolicy` object by running the following command:
42+
+
43+
[source,terminal]
44+
----
45+
$ oc create -f sriovnetpolicy-810-sriov-node-network.yaml
46+
----
47+
48+
. Open a separate terminal window and monitor the synchronization status of the SR-IOV network node state for the node specified in the `openshift-sriov-network-operator` namespace by running the following command:
49+
+
50+
[source,terminal]
51+
----
52+
$ watch -n 1 'oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath="{.status.syncStatus}"'
53+
----
54+
+
55+
The synchronization status indicates a change from `InProgress` to `Succeeded`.
56+
57+
. Create a `SriovNetwork` object, and set the outer VLAN called the S-tag, or `Service Tag`, as it belongs to the infrastructure.
58+
+
59+
[IMPORTANT]
60+
====
61+
You must configure the VLAN on the trunk interface of the switch. In addition, you might need to further configure some switches to support QinQ tagging.
62+
====
63+
64+
.. Create a file named `nad-sriovnetwork-1ad-810.yaml` by using the following content:
65+
+
66+
[source,yaml]
67+
----
68+
apiVersion: sriovnetwork.openshift.io/v1
69+
kind: SriovNetwork
70+
metadata:
71+
name: sriovnetwork-1ad-810
72+
namespace: openshift-sriov-network-operator
73+
spec:
74+
ipam: '{}'
75+
vlan: 171 <1>
76+
vlanProto: "802.1ad" <2>
77+
networkNamespace: default
78+
resourceName: resource810
79+
----
80+
+
81+
<1> Sets the S-tag VLAN tag to `171`.
82+
<2> Specifies the VLAN protocol to assign to the virtual function (VF). Supported values are `802.1ad` and `802.1q`. The default value is `802.1q`.
83+
84+
.. Create the object by running the following command:
85+
+
86+
[source,terminal]
87+
----
88+
$ oc create -f nad-sriovnetwork-1ad-810.yaml
89+
----
90+
91+
. Create a `NetworkAttachmentDefinition` object with an inner VLAN. The inner VLAN is often referred to as the C-tag, or `Customer Tag`, as it belongs to the Network Function:
92+
93+
.. Create a file named `nad-cvlan100.yaml` by using the following content:
94+
+
95+
[source,yaml]
96+
----
97+
apiVersion: k8s.cni.cncf.io/v1
98+
kind: NetworkAttachmentDefinition
99+
metadata:
100+
name: nad-cvlan100
101+
namespace: default
102+
spec:
103+
config: '{
104+
"name": "vlan-100",
105+
"cniVersion": "0.3.1",
106+
"type": "vlan",
107+
"linkInContainer": true,
108+
"master": "net1", <1>
109+
"vlanId": 100,
110+
"ipam": {"type": "static"}
111+
}'
112+
----
113+
+
114+
<1> Specifies the VF interface inside the pod. The default name is `net1` as the name is not set in the pod annotation.
115+
116+
.. Apply the YAML file by running the following command:
117+
+
118+
[source,terminal]
119+
----
120+
$ oc apply -f nad-cvlan100.yaml
121+
----
122+
123+
.Verification
124+
125+
* Verify QinQ is active on the node by following this procedure:
126+
127+
. Create a file named `test-qinq-pod.yaml` by using the following content:
128+
+
129+
[source,yaml]
130+
----
131+
apiVersion: v1
132+
kind: Pod
133+
metadata:
134+
name: test-pod
135+
annotations:
136+
k8s.v1.cni.cncf.io/networks: sriovnetwork-1ad-810, nad-cvlan100
137+
spec:
138+
containers:
139+
- name: test-container
140+
image: quay.io/ocp-edge-qe/cnf-gotests-client:v4.10
141+
imagePullPolicy: Always
142+
securityContext:
143+
privileged: true
144+
----
145+
146+
. Create the test pod by running the following command:
147+
+
148+
[source,terminal]
149+
----
150+
$ oc create -f test-qinq-pod.yaml
151+
----
152+
153+
. Enter into a debug session on the target node where the pod is present and display information about the network interface `ens5f0` by running the following command:
154+
+
155+
[source,terminal]
156+
----
157+
$ oc debug node/my-cluster-node -- bash -c "ip link show ens5f0"
158+
----
159+
+
160+
.Example output
161+
162+
[source,terminal]
163+
----
164+
6: ens5f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
165+
link/ether b4:96:91:a5:22:10 brd ff:ff:ff:ff:ff:ff
166+
vf 0 link/ether a2:81:ba:d0:6f:f3 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
167+
vf 1 link/ether 8a:bb:0a:36:f2:ed brd ff:ff:ff:ff:ff:ff, vlan 171, vlan protocol 802.1ad, spoof checking on, link-state auto, trust off
168+
vf 2 link/ether ca:0e:e1:5b:0c:d2 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
169+
vf 3 link/ether ee:6c:e2:f5:2c:70 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
170+
vf 4 link/ether 0a:d6:b7:66:5e:e8 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
171+
vf 5 link/ether da:d5:e7:14:4f:aa brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
172+
vf 6 link/ether d6:8e:85:75:12:5c brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
173+
vf 7 link/ether d6:eb:ce:9c:ea:78 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
174+
vf 8 link/ether 5e:c5:cc:05:93:3c brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
175+
vf 9 link/ether a6:5a:7c:1c:2a:16 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
176+
----
177+
+
178+
The `vlan protocol 802.1ad` ID in the output indicates that the interface supports VLAN tagging with protocol 802.1ad (QinQ). The VLAN ID is 171.

‎modules/nw-sriov-about-qinq.adoc

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
// Module included in the following assembly:
2+
//
3+
// * networking/hardware_networks/configuring-sriov-qinq-support.adocance/configuring-sriov-qinq-support.adoc
4+
5+
6+
:_mod-docs-content-type: CONCEPT
7+
[id="nw-about-qinq-support_{context}"]
8+
= About 802.1Q-in-802.1Q support
9+
10+
In traditional VLAN setups, frames typically contain a single VLAN tag, such as VLAN-100, as well as other metadata such as Quality of Service (QoS) bits and protocol information. QinQ introduces a second VLAN tag, where the service provider designates the outer tag for their use, offering them flexibility, while the inner tag remains dedicated to the customer's VLAN.
11+
12+
QinQ facilitates the creation of nested VLANs by using double VLAN tagging, enabling finer segmentation and isolation of traffic within a network environment. This approach is particularly valuable in service provider networks where you need to deliver VLAN-based services to multiple customers over a common infrastructure, while ensuring separation and isolation of traffic.
13+
14+
When two VLAN tags are present in a packet, the outer VLAN tag can be either 802.1Q or 802.1ad. The inner VLAN tag must always be 802.1Q.
15+
16+
The {product-title} SR-IOV solution already supports setting the VLAN protocol on the `SriovNetwork` custom resource (CR). The virtual function (VF) can use this protocol to set the VLAN tag, also known as the outer tag. Pods can then use the VLAN CNI plugin to configure the inner tag.
17+
18+
.Supported network interface cards
19+
[cols="30%,30%,40%",options="header"]
20+
|===
21+
| NIC | 802.1ad/802.1Q | 802.1Q/802.1Q
22+
23+
| Intel X710 | No
24+
a|Supported
25+
26+
| Intel E810 | Supported
27+
a| Supported
28+
29+
| Mellanox | No
30+
a| Supported
31+
|===

‎networking/hardware_networks/configuring-sriov-qinq-support.adoc

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
[id="configuring-qinq-support"]
3+
= Configuring QinQ support for SR-IOV enabled workloads
4+
include::_attributes/common-attributes.adoc[]
5+
:context: configuring-qinq-support
6+
7+
toc::[]
8+
9+
QinQ, formally known as 802.1Q-in-802.1Q, is a networking technique defined by IEEE 802.1ad. IEEE 802.1ad extends the IEEE 802.1Q-1998 standard and enriches VLAN capabilities by introducing an additional 802.1Q tag to packets already tagged with 802.1Q. This method is also referred to as VLAN stacking or double VLAN.
10+
11+
include::modules/nw-sriov-about-qinq.adoc[leveloffset=+1]
12+
13+
[role="_additional-resources"]
14+
.Additional resources
15+
16+
* xref:../../networking/multiple_networks/configuring-additional-network.adoc#nw-multus-vlan-object_configuring-additional-network[Configuration for an VLAN additional network]
17+
18+
include::modules/nw-configuring-qinq-sriov-proc.adoc[leveloffset=+1]

0 commit comments

Comments
 (0)