Skip to content

Commit 07b4f7a

Browse files
sheriff-rhsheriff-rh
authored
Merge pull request #78839 from dfitzmau/OCPBUGS-34611-new
2 parents e2bf5a2 + 2f084e8 commit 07b4f7a

File tree

2 files changed

+304
-0
lines changed

2 files changed

+304
-0
lines changed

‎modules/configuring-ovnk-use-second-ovs-bridge.adoc

Lines changed: 296 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,296 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * support/troubleshooting/troubleshooting-network-issues.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="configuring-ovnk-use-second-ovs-bridge_{context}"]
7+
= Configuring OVN-Kubernetes to use a secondary OVS bridge
8+
9+
You can create an additional or _secondary_ Open vSwitch (OVS) bridge, `br-ex1`, that OVN-Kubernetes manages and the Multiple External Gateways (MEG) implementation uses for defining external gateways for an {product-title} node. You can define a MEG in an `AdminPolicyBasedExternalRoute` custom resource (CR). The MEG implementation provides a pod with access to multiple gateways, equal-cost multipath (ECMP) routes, and the Bidirectional Forwarding Detection (BFD) implementation.
10+
11+
Consider a use case for pods impacted by the Multiple External Gateways (MEG) feature and you want to egress traffic to a different interface, for example `br-ex1`, on a node. Egress traffic for pods not impacted by MEG get routed to the default OVS `br-ex` bridge.
12+
13+
You must define the additional bridge in an interface definition of a machine configuration manifest file. The Machine Config Operator uses the manifest to create a new file at `/etc/ovnk/extra_bridge` on the host. The new file includes the name of the network interface that the additional OVS bridge configures for a node.
14+
15+
After you create and edit the manifest file, the Machine Config Operator completes tasks in the following order:
16+
17+
. Drains nodes in singular order based on the selected machine configuration pool.
18+
. Injects Ignition configuration files into each node, so that each node receives the additional `br-ex1` bridge network configuration.
19+
. Verify that the `br-ex` MAC address matches the MAC address for the interface that `br-ex` uses for the network connection.
20+
. Executes the `configure-ovs.sh` shell script that references the new interface definition.
21+
. Adds `br-ex` and `br-ex1` to the host node.
22+
. Uncordons the nodes.
23+
24+
[NOTE]
25+
====
26+
After all the nodes return to the `Ready` state and the OVN-Kubernetes Operator detects and configures `br-ex` and `br-ex1`, the Operator applies the `k8s.ovn.org/l3-gateway-config` annotation to each node.
27+
====
28+
29+
For more information about useful situations for the additional `br-ex1` bridge and a situation that always requires the default `br-ex` bridge, see "Configuration for a localnet topology".
30+
31+
.Procedure
32+
33+
. Optional: Create an interface connection that your additional bridge, `br-ex1`, can use by completing the following steps. The example steps show the creation of a new bond and its dependent interfaces that are all defined in a machine configuration manifest file. The additional bridge uses the `MachineConfig` object to form a additional bond interface.
34+
+
35+
[IMPORTANT]
36+
====
37+
Do not use the Kubernetes NMState Operator to define or a `NodeNetworkConfigurationPolicy` (NNCP) manifest file to define the additional interface.
38+
39+
Also ensure that the additional interface or sub-interfaces when defining a `bond` interface are not used by an existing `br-ex` OVN Kubernetes network deployment.
40+
====
41+
+
42+
.. Create the following interface definition files. These files get added to a machine configuration manifest file so that host nodes can access the definition files.
43+
+
44+
.Example of the first interface definition file that is named `eno1.config`
45+
[source,ini]
46+
----
47+
[connection]
48+
id=eno1
49+
type=ethernet
50+
interface-name=eno1
51+
master=bond1
52+
slave-type=bond
53+
autoconnect=true
54+
autoconnect-priority=20
55+
----
56+
+
57+
.Example of the second interface definition file that is named `eno2.config`
58+
[source,ini]
59+
----
60+
[connection]
61+
id=eno2
62+
type=ethernet
63+
interface-name=eno2
64+
master=bond1
65+
slave-type=bond
66+
autoconnect=true
67+
autoconnect-priority=20
68+
----
69+
+
70+
.Example of the second bond interface definition file that is named `bond1.config`
71+
[source,ini]
72+
----
73+
[connection]
74+
id=bond1
75+
type=bond
76+
interface-name=bond1
77+
autoconnect=true
78+
connection.autoconnect-slaves=1
79+
autoconnect-priority=20
80+
81+
[bond]
82+
mode=802.3ad
83+
miimon=100
84+
xmit_hash_policy="layer3+4"
85+
86+
[ipv4]
87+
method=auto
88+
----
89+
+
90+
.. Convert the definition files to Base64 encoded strings by running the following command:
91+
+
92+
[source,terminal]
93+
----
94+
$ base64 <directory_path>/en01.config
95+
----
96+
+
97+
[source,terminal]
98+
----
99+
$ base64 <directory_path>/eno2.config
100+
----
101+
+
102+
[source,terminal]
103+
----
104+
$ base64 <directory_path>/bond1.config
105+
----
106+
107+
. Prepare the environment variables. Replace `<machine_role>` with the node role, such as `worker`, and replace `<interface_name>` with the name of your additional `br-ex` bridge name.
108+
+
109+
[source,terminal]
110+
----
111+
$ export ROLE=<machine_role>
112+
----
113+
114+
. Define each interface definition in a machine configuration manifest file:
115+
+
116+
.Example of a machine configuration file with definitions added for `bond1`, `eno1`, and `en02`
117+
[source,yaml]
118+
----
119+
apiVersion: machineconfiguration.openshift.io/v1
120+
kind: MachineConfig
121+
metadata:
122+
labels:
123+
machineconfiguration.openshift.io/role: ${worker}
124+
name: 12-${ROLE}-sec-bridge-cni
125+
spec:
126+
config:
127+
ignition:
128+
version: 3.2.0
129+
storage:
130+
files:
131+
- contents:
132+
source: data:;base64,<base-64-encoded-contents-for-bond1.conf>
133+
path: /etc/NetworkManager/system-connections/bond1.nmconnection
134+
filesystem: root
135+
mode: 0600
136+
- contents:
137+
source: data:;base64,<base-64-encoded-contents-for-eno1.conf>
138+
path: /etc/NetworkManager/system-connections/eno1.nmconnection
139+
filesystem: root
140+
mode: 0600
141+
- contents:
142+
source: data:;base64,<base-64-encoded-contents-for-eno2.conf>
143+
path: /etc/NetworkManager/system-connections/eno2.nmconnection
144+
filesystem: root
145+
mode: 0600
146+
# ...
147+
----
148+
149+
. Create a machine configuration manifest file for configuring the network plugin by entering the following command in your terminal:
150+
+
151+
[source,terminal]
152+
----
153+
$ oc create -f <machine_config_file_name>
154+
----
155+
156+
. Create an Open vSwitch (OVS) bridge, `br-ex1`, on nodes by using the OVN-Kubernetes network plugin to create an `extra_bridge` file`. Ensure that you save the file in the `/etc/ovnk/extra_bridge` path of the host. The file must state the interface name that supports the additional bridge and not the default interface that supports `br-ex`, which holds the primary IP address of the node.
157+
+
158+
.Example configuration for the `extra_bridge` file, `/etc/ovnk/extra_bridge`, that references a additional interface
159+
[source,text]
160+
----
161+
bond1
162+
----
163+
164+
. Create a machine configuration manifest file that defines the existing static interface that hosts `br-ex1` on any nodes restarted on your cluster:
165+
+
166+
.Example of a machine configuration file that defines `bond1` as the interface for hosting `br-ex1`
167+
[source,yaml]
168+
----
169+
apiVersion: machineconfiguration.openshift.io/v1
170+
kind: MachineConfig
171+
metadata:
172+
labels:
173+
machineconfiguration.openshift.io/role: ${worker}
174+
name: 12-worker-extra-bridge
175+
spec:
176+
config:
177+
ignition:
178+
version: 3.2.0
179+
storage:
180+
files:
181+
- path: /etc/ovnk/extra_bridge
182+
mode: 0420
183+
overwrite: true
184+
contents:
185+
source: data:text/plain;charset=utf-8,bond1
186+
filesystem: root
187+
----
188+
189+
. Apply the machine-configuration to your selected nodes:
190+
+
191+
[source,terminal]
192+
----
193+
$ oc create -f <machine_config_file_name>
194+
----
195+
196+
. Optional: You can override the `br-ex` selection logic for nodes by creating a machine configuration file that in turn creates a `/var/lib/ovnk/iface_default_hint` resource.
197+
+
198+
[NOTE]
199+
====
200+
The resource lists the name of the interface that `br-ex` selects for your cluster. By default, `br-ex` selects the primary interface for a node based on boot order and the IP address subnet in the machine network. Certain machine network configurations might require that `br-ex` continues to select the default interfaces or bonds for a host node.
201+
====
202+
+
203+
.. Create a machine configuration file on the host node to override the default interface.
204+
+
205+
[IMPORTANT]
206+
====
207+
Only create this machine configuration file for the purposes of changing the `br-ex` selection logic. Using this file to change the IP addresses of existing nodes in your cluster is not supported.
208+
====
209+
+
210+
.Example of a machine configuration file that overrides the default interface
211+
[source,yaml]
212+
----
213+
apiVersion: machineconfiguration.openshift.io/v1
214+
kind: MachineConfig
215+
metadata:
216+
labels:
217+
machineconfiguration.openshift.io/role: ${worker}
218+
name: 12-worker-br-ex-override
219+
spec:
220+
config:
221+
ignition:
222+
version: 3.2.0
223+
storage:
224+
files:
225+
- path: /var/lib/ovnk/iface_default_hint
226+
mode: 0420
227+
overwrite: true
228+
contents:
229+
source: data:text/plain;charset=utf-8,bond0 <1>
230+
filesystem: root
231+
----
232+
<1> Ensure `bond0` exists on the node before you apply the machine configuration file to the node.
233+
+
234+
.. Before you apply the configuration to all new nodes in your cluster, reboot the host node to verify that `br-ex` selects the intended interface and does not conflict with the new interfaces that you defined on `br-ex1`.
235+
+
236+
.. Apply the machine configuration file to all new nodes in your cluster:
237+
+
238+
[source,terminal]
239+
----
240+
$ oc create -f <machine_config_file_name>
241+
----
242+
243+
.Verification
244+
245+
. Identify the IP addresses of nodes with the `exgw-ip-addresses` label in your cluster to verify that the nodes use the additional bridge instead of the default bridge:
246+
+
247+
[source,terminal]
248+
----
249+
$ oc get nodes -o json | grep --color exgw-ip-addresses
250+
----
251+
+
252+
.Example output
253+
[source,terminal]
254+
----
255+
"k8s.ovn.org/l3-gateway-config":
256+
\"exgw-ip-address\":\"172.xx.xx.yy/24\",\"next-hops\":[\"xx.xx.xx.xx\"],
257+
----
258+
259+
. Observe that the additional bridge exists on target nodes by reviewing the network interface names on the host node:
260+
+
261+
[source,terminal]
262+
----
263+
$ oc debug node/<node_name> -- chroot /host sh -c "ip a | grep mtu | grep br-ex"
264+
----
265+
+
266+
.Example output
267+
[source,terminal]
268+
----
269+
Starting pod/worker-1-debug ...
270+
To use host binaries, run `chroot /host`
271+
# ...
272+
5: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
273+
6: br-ex1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
274+
----
275+
276+
. Optional: If you use `/var/lib/ovnk/iface_default_hint`, check that the MAC address of `br-ex` matches the MAC address of the primary selected interface:
277+
+
278+
[source,terminal]
279+
----
280+
$ oc debug node/<node_name> -- chroot /host sh -c "ip a | grep -A1 -E 'br-ex|bond0'
281+
----
282+
+
283+
.Example output that shows the primary interface for `br-ex` as `bond0`
284+
[source,terminal]
285+
----
286+
Starting pod/worker-1-debug ...
287+
To use host binaries, run `chroot /host`
288+
# ...
289+
sh-5.1# ip a | grep -A1 -E 'br-ex|bond0'
290+
2: bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
291+
link/ether fa:16:3e:47:99:98 brd ff:ff:ff:ff:ff:ff
292+
--
293+
5: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
294+
link/ether fa:16:3e:47:99:98 brd ff:ff:ff:ff:ff:ff
295+
inet 10.xx.xx.xx/21 brd 10.xx.xx.255 scope global dynamic noprefixroute br-ex
296+
----

‎support/troubleshooting/troubleshooting-network-issues.adoc

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,14 @@ include::modules/nw-how-nw-iface-selected.adoc[leveloffset=+1]
1111

1212
include::modules/overriding-default-node-ip-selection-logic.adoc[leveloffset=+2]
1313

14+
// Configuring OVN-Kubernetes to use a secondary OVS bridge
15+
include::modules/configuring-ovnk-use-second-ovs-bridge.adoc[leveloffset=+2]
16+
17+
[role="_additional-resources"]
18+
.Additional resources
19+
20+
* xref:../../networking/ovn_kubernetes_network_provider/configuring-secondary-external-gateway.adoc#configuring-secondary-external-gateway[Configure an external gateway on the default network]
21+
1422
// Troubleshooting OVS issues
1523
include::modules/nw-troubleshoot-ovs.adoc[leveloffset=+1]
1624

0 commit comments

Comments
 (0)