OSDOCS-7075: Documented support for multiple subnets

Author: dfitzmau

_topic_maps/_topic_map.yml

@@ -333,22 +333,6 @@ Topics:
     File: uninstalling-cluster-nutanix
   - Name: Installation configuration parameters for Nutanix
     File: installation-config-parameters-nutanix
-- Name: Installing on bare metal
-  Dir: installing_bare_metal
-  Distros: openshift-origin,openshift-enterprise
-  Topics:
-  - Name: Preparing to install on bare metal
-    File: preparing-to-install-on-bare-metal
-  - Name: Installing a user-provisioned cluster on bare metal
-    File: installing-bare-metal
-  - Name: Installing a user-provisioned bare metal cluster with network customizations
-    File: installing-bare-metal-network-customizations
-  - Name: Installing a user-provisioned bare metal cluster on a restricted network
-    File: installing-restricted-networks-bare-metal
-  - Name: Scaling a user-provisioned installation with the bare metal operator
-    File: scaling-a-user-provisioned-cluster-with-the-bare-metal-operator
-  - Name: Installation configuration parameters for bare metal
-    File: installation-config-parameters-bare-metal
 - Name: Installing on-premise with Assisted Installer
   Dir: installing_on_prem_assisted
   Distros: openshift-enterprise
@@ -379,6 +363,22 @@ Topics:
     File: install-sno-preparing-to-install-sno
   - Name: Installing OpenShift on a single node
     File: install-sno-installing-sno
+- Name: Installing on bare metal
+  Dir: installing_bare_metal
+  Distros: openshift-origin,openshift-enterprise
+  Topics:
+  - Name: Preparing to install on bare metal
+    File: preparing-to-install-on-bare-metal
+  - Name: Installing a user-provisioned cluster on bare metal
+    File: installing-bare-metal
+  - Name: Installing a user-provisioned bare metal cluster with network customizations
+    File: installing-bare-metal-network-customizations
+  - Name: Installing a user-provisioned bare metal cluster on a restricted network
+    File: installing-restricted-networks-bare-metal
+  - Name: Scaling a user-provisioned installation with the bare metal operator
+    File: scaling-a-user-provisioned-cluster-with-the-bare-metal-operator
+  - Name: Installation configuration parameters for bare metal
+    File: installation-config-parameters-bare-metal
 - Name: Deploying installer-provisioned clusters on bare metal
   Dir: installing_bare_metal_ipi
   Distros: openshift-origin,openshift-enterprise

installing/installing_bare_metal_ipi/ipi-install-installation-workflow.adoc

@@ -21,6 +21,7 @@ include::modules/ipi-install-checking-ntp-sync.adoc[leveloffset=+1]
 
 include::modules/ipi-install-configuring-networking.adoc[leveloffset=+1]
 
+// Establishing communication between subnets
 include::modules/ipi-install-establishing-communication-between-subnets.adoc[leveloffset=+1]
 
 include::modules/ipi-install-retrieving-the-openshift-installer.adoc[leveloffset=+1]

modules/ipi-install-configuring-host-network-interfaces-for-subnets.adoc

@@ -6,15 +6,18 @@
 [id="ipi-install-configuring-host-network-interfaces-for-subnets_{context}"]
 = Configuring host network interfaces for subnets
 
-For edge computing scenarios, it can be beneficial to locate worker nodes closer to the edge. To locate remote worker nodes in subnets, you might use different network segments or subnets for the remote worker nodes than you used for the control plane subnet and local worker nodes. You can reduce latency for the edge and allow for enhanced scalability by setting up subnets for edge computing scenarios.
-
-If you have established different network segments or subnets for remote worker nodes as described in the section on "Establishing communication between subnets", you must specify the subnets in the `machineNetwork` configuration setting if the workers are using static IP addresses, bonds or other advanced networking. When setting the node IP address in the `networkConfig` parameter for each remote worker node, you must also specify the gateway and the DNS server for the subnet containing the control plane nodes when using static IP addresses. This ensures the remote worker nodes can reach the subnet containing the control plane nodes and that they can receive network traffic from the control plane.
+For edge computing scenarios, it can be beneficial to locate compute nodes closer to the edge. To locate remote nodes in subnets, you might use different network segments or subnets for the remote nodes than you used for the control plane subnet and local compute nodes. You can reduce latency for the edge and allow for enhanced scalability by setting up subnets for edge computing scenarios.
 
 [IMPORTANT]
 ====
-All control plane nodes must run in the same subnet. When using more than one subnet, you can also configure the Ingress VIP to run on the control plane nodes by using a manifest. See "Configuring network components to run on the control plane" for details.
+When using the default load balancer, `OpenShiftManagedDefault` and adding remote nodes to your {product-title} cluster, all control plane nodes must run in the same subnet. When using more than one subnet, you can also configure the Ingress VIP to run on the control plane nodes by using a manifest. See "Configuring network components to run on the control plane" for details.
+====
 
-Deploying a cluster with multiple subnets requires using virtual media, such as `redfish-virtualmedia` and `idrac-virtualmedia`.
+If you have established different network segments or subnets for remote nodes as described in the section on "Establishing communication between subnets", you must specify the subnets in the `machineNetwork` configuration setting if the workers are using static IP addresses, bonds or other advanced networking. When setting the node IP address in the `networkConfig` parameter for each remote node, you must also specify the gateway and the DNS server for the subnet containing the control plane nodes when using static IP addresses. This ensures the remote nodes can reach the subnet containing the control plane nodes and that they can receive network traffic from the control plane.
+
+[NOTE]
+====
+Deploying a cluster with multiple subnets requires using virtual media, such as `redfish-virtualmedia` and `idrac-virtualmedia`, because remote nodes cannot access the local provisioning network.
 ====
 
 .Procedure

modules/ipi-install-establishing-communication-between-subnets.adoc

@@ -6,21 +6,29 @@
 [id="ipi-install-establishing-communication-between-subnets_{context}"]
 = Establishing communication between subnets
 
-In a typical {product-title} cluster setup, all nodes, including the control plane and worker nodes, reside in the same network. However, for edge computing scenarios, it can be beneficial to locate worker nodes closer to the edge. This often involves using different network segments or subnets for the remote worker nodes than the subnet used by the control plane and local worker nodes. Such a setup can reduce latency for the edge and allow for enhanced scalability. However, the network must be configured properly before installing {product-title} to ensure that the edge subnets containing the remote worker nodes can reach the subnet containing the control plane nodes and receive traffic from the control plane too.
+In a typical {product-title} cluster setup, all nodes, including the control plane and compute nodes, reside in the same network. However, for edge computing scenarios, it can be beneficial to locate compute nodes closer to the edge. This often involves using different network segments or subnets for the remote nodes than the subnet used by the control plane and local compute nodes. Such a setup can reduce latency for the edge and allow for enhanced scalability.
 
-[IMPORTANT]
-====
-All control plane nodes must run in the same subnet. When using more than one subnet, you can also configure the Ingress VIP to run on the control plane nodes by using a manifest. See "Configuring network components to run on the control plane" for details.
+Before installing {product-title}, you must configure the network properly to ensure that the edge subnets containing the remote nodes can reach the subnet containing the control plane nodes and receive traffic from the control plane too.
+
+You can run control plane nodes in the same subnet or multiple subnets by configuring a user-managed load balancer in place of the default load balancer. With a multiple subnet environment, you can reduce the risk of your {product-title} cluster from failing because of a hardware failure or a network outage. For more information, see "Services for a user-managed load balancer" and "Configuring a user-managed load balancer".
+
+Running control plane nodes in a multiple subnet environment requires completion of the following key tasks:
 
-Deploying a cluster with multiple subnets requires using virtual media.
+* Configuring a user-managed load balancer instead of the default load balancer by specifying `UserManaged` in the `loadBalancer.type` parameter of the `install-config.yaml` file.
+* Configuring a user-managed load balancer address in the `ingressVIPs` and `apiVIPs` parameters of the `install-config.yaml` file.
+* Adding the multiple subnet Classless Inter-Domain Routing (CIDR) and the user-managed load balancer IP addresses to the `networking.machineNetworks` parameter in the `install-config.yaml` file.
+
+[NOTE]
+====
+Deploying a cluster with multiple subnets requires using virtual media, such as `redfish-virtualmedia` and `idrac-virtualmedia`.
 ====
 
-This procedure details the network configuration required to allow the remote worker nodes in the second subnet to communicate effectively with the control plane nodes in the first subnet and to allow the control plane nodes in the first subnet to communicate effectively with the remote worker nodes in the second subnet.
+The procedure details the network configuration required to allow the remote nodes in the second subnet to communicate effectively with the control plane nodes in the first subnet and to allow the control plane nodes in the first subnet to communicate effectively with the remote nodes in the second subnet.
 
 In this procedure, the cluster spans two subnets:
 
-- The first subnet (`10.0.0.0`) contains the control plane and local worker nodes.
-- The second subnet (`192.168.0.0`) contains the edge worker nodes.
+- The first subnet (`10.0.0.0`) contains the control plane and local compute nodes.
+- The second subnet (`192.168.0.0`) contains the edge compute nodes.
 
 .Procedure
 
@@ -134,22 +142,22 @@ Replace `<interface_name>` with the interface name.
 Adjust the commands to match your actual interface names and gateway.
 ====
 
-. Once you have configured the networks, test the connectivity to ensure the remote worker nodes can reach the control plane nodes and the control plane nodes can reach the remote worker nodes.
+. After you have configured the networks, test the connectivity to ensure the remote nodes can reach the control plane nodes and the control plane nodes can reach the remote nodes.
 
-.. From the control plane nodes in the first subnet, ping a remote worker node in the second subnet by running the following command:
+.. From the control plane nodes in the first subnet, ping a remote node in the second subnet by running the following command:
 +
 [source,terminal]
 ----
-$ ping <remote_worker_node_ip_address>
+$ ping <remote_node_ip_address>
 ----
 +
-If the ping is successful, it means the control plane nodes in the first subnet can reach the remote worker nodes in the second subnet. If you do not receive a response, review the network configurations and repeat the procedure for the node.
+If the ping is successful, it means the control plane nodes in the first subnet can reach the remote nodes in the second subnet. If you do not receive a response, review the network configurations and repeat the procedure for the node.
 
-.. From the remote worker nodes in the second subnet, ping a control plane node in the first subnet by running the following command:
+.. From the remote nodes in the second subnet, ping a control plane node in the first subnet by running the following command:
 +
 [source,terminal]
 ----
 $ ping <control_plane_node_ip_address>
 ----
 +
-If the ping is successful, it means the remote worker nodes in the second subnet can reach the control plane in the first subnet. If you do not receive a response, review the network configurations and repeat the procedure for the node.
+If the ping is successful, it means the remote nodes in the second subnet can reach the control plane in the first subnet. If you do not receive a response, review the network configurations and repeat the procedure for the node.

modules/nw-osp-configuring-external-load-balancer.adoc

@@ -95,12 +95,12 @@ Interval: 10
 
 .Procedure
 
-. Configure the HAProxy Ingress Controller, so that you can enable access to the cluster from your load balancer on ports 6443, 443, and 80:
+. Configure the HAProxy Ingress Controller, so that you can enable access to the cluster from your load balancer on ports 6443, 22623, 443, and 80. Depending on your needs, you can specify the IP address of a single subnet or IP addresses from multiple subnets in your HAProxy configuration.
 +
-.Example HAProxy configuration
-[source,terminal]
+.Example HAProxy configuration with one listed subnet
+[source,terminal,subs="quotes"]
 ----
-#...
+# ...
 listen my-cluster-api-6443
     bind 192.168.1.100:6443
     mode tcp
@@ -126,28 +126,82 @@ listen my-cluster-machine-config-api-22623
     server my-cluster-master-1 192.168.1.103:22623 check inter 10s rise 2 fall 2
 
 listen my-cluster-apps-443
-        bind 192.168.1.100:443
-        mode tcp
-        balance roundrobin
-    option httpchk
-    http-check connect
-    http-check send meth GET uri /healthz/ready
-    http-check expect status 200
-        server my-cluster-worker-0 192.168.1.111:443 check port 1936 inter 10s rise 2 fall 2
-        server my-cluster-worker-1 192.168.1.112:443 check port 1936 inter 10s rise 2 fall 2
-        server my-cluster-worker-2 192.168.1.113:443 check port 1936 inter 10s rise 2 fall 2
+    bind 192.168.1.100:443
+    mode tcp
+    balance roundrobin
+  option httpchk
+  http-check connect
+  http-check send meth GET uri /healthz/ready
+  http-check expect status 200
+    server my-cluster-worker-0 192.168.1.111:443 check port 1936 inter 10s rise 2 fall 2
+    server my-cluster-worker-1 192.168.1.112:443 check port 1936 inter 10s rise 2 fall 2
+    server my-cluster-worker-2 192.168.1.113:443 check port 1936 inter 10s rise 2 fall 2
 
 listen my-cluster-apps-80
-        bind 192.168.1.100:80
-        mode tcp
-        balance roundrobin
-    option httpchk
-    http-check connect
-    http-check send meth GET uri /healthz/ready
-    http-check expect status 200
-        server my-cluster-worker-0 192.168.1.111:80 check port 1936 inter 10s rise 2 fall 2
-        server my-cluster-worker-1 192.168.1.112:80 check port 1936 inter 10s rise 2 fall 2
-        server my-cluster-worker-2 192.168.1.113:80 check port 1936 inter 10s rise 2 fall 2
+   bind 192.168.1.100:80
+   mode tcp
+   balance roundrobin
+  option httpchk
+  http-check connect
+  http-check send meth GET uri /healthz/ready
+  http-check expect status 200
+    server my-cluster-worker-0 192.168.1.111:80 check port 1936 inter 10s rise 2 fall 2
+    server my-cluster-worker-1 192.168.1.112:80 check port 1936 inter 10s rise 2 fall 2
+    server my-cluster-worker-2 192.168.1.113:80 check port 1936 inter 10s rise 2 fall 2
+# ...
+----
++
+.Example HAProxy configuration with multiple listed subnets
+[source,terminal,subs="quotes"]
+----
+# ...
+listen api-server-6443
+    bind *:6443
+    mode tcp
+      server master-00 192.168.83.89:6443 check inter 1s
+      server master-01 192.168.84.90:6443 check inter 1s
+      server master-02 192.168.85.99:6443 check inter 1s
+      server bootstrap 192.168.80.89:6443 check inter 1s
+
+listen machine-config-server-22623
+    bind *:22623
+    mode tcp
+      server master-00 192.168.83.89:22623 check inter 1s
+      server master-01 192.168.84.90:22623 check inter 1s
+      server master-02 192.168.85.99:22623 check inter 1s
+      server bootstrap 192.168.80.89:22623 check inter 1s
+
+listen ingress-router-80
+    bind *:80
+    mode tcp
+    balance source
+      server worker-00 192.168.83.100:80 check inter 1s
+      server worker-01 192.168.83.101:80 check inter 1s
+
+listen ingress-router-443
+    bind *:443
+    mode tcp
+    balance source
+      server worker-00 192.168.83.100:443 check inter 1s
+      server worker-01 192.168.83.101:443 check inter 1s
+
+listen ironic-api-6385
+    bind *:6385
+    mode tcp
+    balance source
+      server master-00 192.168.83.89:6385 check inter 1s
+      server master-01 192.168.84.90:6385 check inter 1s
+      server master-02 192.168.85.99:6385 check inter 1s
+      server bootstrap 192.168.80.89:6385 check inter 1s
+
+listen inspector-api-5050
+    bind *:5050
+    mode tcp
+    balance source
+      server master-00 192.168.83.89:5050 check inter 1s
+      server master-01 192.168.84.90:5050 check inter 1s
+      server master-02 192.168.85.99:5050 check inter 1s
+      server bootstrap 192.168.80.89:5050 check inter 1s
 # ...
 ----