Support for RoCE (Rdma over converged Ethernet) using Broadcom 200G Nics

[viswar4]

Hello all,

We are evaluating Talos Linux for our bare metal servers. We need to setup GPU direct using RoCE. We have Broadcom Thor 2 Nics. Is this possible? What would be the requirements/pre-requisites for such a setup.

[rothgar]

Do the broadcom nics show up in talos? If not the first step would be to create a package to add drivers to the talos image. Once the nic is working I have no idea how a GPU would show up on a Linux host (does it show up in /dev?) but you would then need to add extensions for the GPU drivers. Here's a video to add a GPU to Talos assuming it shows up like a regular PCI device. https://youtu.be/HiDWGs1PYhc

[viswar4]

Hi,

I have not actually set up the cluster. Wanted to see if it is even possible. The GPUs are Nvidia H200. Will try this out and update.