OCPBUGS-48847: Documented IPoIB support for nmstate

Author: dfitzmau

installing/installing_bare_metal/ipi/ipi-install-post-installation-configuration.adoc

@@ -17,6 +17,9 @@ include::modules/nw-enabling-a-provisioning-network-after-installation.adoc[leve
 // Creating a manifest object that includes a customized `br-ex` bridge
 include::modules/creating-manifest-file-customized-br-ex-bridge.adoc[leveloffset=+1]
 
+// Creating an InfiniBand interface on nodes
+include::modules/virt-creating-infiniband-interface-on-nodes.adoc[leveloffset=+1]
+
 // Services for a user-managed load balancer
 include::modules/nw-osp-services-external-load-balancer.adoc[leveloffset=+1]
 

modules/creating-manifest-file-customized-br-ex-bridge.adoc

@@ -17,7 +17,7 @@ endif::[]
 
 :_mod-docs-content-type: PROCEDURE
 [id="creating-manifest-file-customized-br-ex-bridge_{context}"]
-== Creating a manifest object that includes a customized `br-ex` bridge
+= Creating a manifest object that includes a customized `br-ex` bridge
 
 ifndef::postinstall-bare-metal-ipi,postinstall-bare-metal-upi[]
 As an alternative to using the `configure-ovs.sh` shell script to set a `br-ex` bridge on a bare-metal platform, you can create a `MachineConfig` object that includes an NMState configuration file. The NMState configuration file creates a customized `br-ex` bridge network configuration on each node in your cluster.

modules/virt-creating-infiniband-interface-on-nodes.adoc

@@ -0,0 +1,64 @@
+// Module included in the following assemblies:
+//
+// * installing/installing_bare_metal/ipi/ipi-install-post-installation-configuration.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="virt-creating-infiniband-interface-on-nodes_{context}"]
+= Creating an IP over InfiniBand interface on nodes
+
+On the {product-title} web console, you can install a Red{nbsp}Hat certified third-party Operator, such as the NVIDIA Network Operator, that supports InfiniBand (IPoIB) mode. Typically, you would use the third-party Operator with other vendor infrastructure to manage resources in an {product-title} cluster. To create an IPoIB interface on nodes in your cluster, you must define an InfiniBand (IPoIB) interface in a `NodeNetworkConfigurationPolicy` (NNCP) manifest file. 
+
+[IMPORTANT]
+====
+The {product-title} documentation describes defining only the IPoIB interface configuration in a `NodeNetworkConfigurationPolicy` (NNCP) manifest file. You must refer to the NVIDIA and other third-party vendor documentation for the majority of the configuring steps. Red{nbsp}Hat support does not extend to anything external to the NNCP configuration. 
+
+For more information about the NVIDIA Operator, see link:https://docs.nvidia.com/networking/display/kubernetes2410/getting+started+with+red+hat+openshift[Getting Started with Red{nbsp}Hat OpenShift] (NVIDIA Docs Hub).
+====
+
+.Prerequisites
+
+* You installed a Red{nbsp}Hat certified third-party Operator that supports an IPoIB interface.
+
+
+.Procedure
+
+. Create or edit a `NodeNetworkConfigurationPolicy` (NNCP) manifest file, and then specify an IPoIB interface in the file.
++
+
+[source,yaml]
+----
+apiVersion: nmstate.io/v1
+kind: NodeNetworkConfigurationPolicy
+metadata:
+  name: worker-0-ipoib
+spec:
+# ...
+    interfaces:
+    - description: ""
+      infiniband:
+        mode: datagram <1>
+        pkey: "0xffff" <2>
+      ipv4:
+        address:
+        - ip: 100.125.3.4
+          prefix-length: 16
+        dhcp: false
+        enabled: true
+      ipv6:
+        enabled: false
+      name: ibp27s0
+      state: up
+      type: infiniband <3>
+# ...
+----
+<1> `datagram` is the default mode for an IPoIB interface, and this mode improves optimizes performance and latency. `connected` mode is a supported mode but consider only using this mode when you need to adjust the maximum transmission unit (MTU) value to improve node connectivity with surrounding network devices.  
+<2> Supports a string or an integer value. The parameter defines the protection key, or _P-key_, for the interface for the purposes of authentication and encrypted communications with a third-party vendor, such as NVIDIA. Values `None` and `0xffff` indicate the protection key for the base interface in an InfiniBand system.
+<3> Sets the type of interface to `infiniband `.
+
+. Apply the NNCP configuration to each node in your cluster by running the following command. The Kubernetes NMState Operator can then create an IPoIB interface on each node. 
++
+[source,yaml]
+----
+$ oc apply -f <nncp_file_name> <1>
+----
+<1> Replace `<nncp_file_name>` with the name of your NNCP file.