Nvidia 50xx with Talos 1.10.3 - Nvidia modules are not found

[flashpixx]

Hello,

I try to make it work that my new computer (one node) can run within the pods my Nvidia 5090.
I follow the NVIDIA GPU (Proprietary drivers) and prepare with the image factory (online / running locally with docker compose) my own ISO bare-metal image. My customizing to generate the ISO is

customization:
    systemExtensions:
        officialExtensions:
            - siderolabs/nonfree-kmod-nvidia-production
            - siderolabs/nvidia-container-toolkit-production

and on the controlplane.yamlI set for installing the system

machine:
    sysctls:
        net.core.bpf_jit_harden: 1

    kernel:
        modules:
            - name: nvidia
            - name: nvidia_uvm
            - name: nvidia_drm
            - name: nvidia_modeset

But I get the error on dmesg the no nvidia model is found. I Have tested it with 1.10.1. 1.10.2, 1.10.3. Did I something wrong?

[rothgar]

What does talosctl get pcidevices show?

[flashpixx]

II have grep-ed the Nvidia parts only:

192.168.20.7   hardware    PCIDevice   0000:01:00.0   1         Display controller          VGA compatible controller        NVIDIA Corporation                       GB202 [GeForce RTX 5090]
192.168.20.7   hardware    PCIDevice   0000:01:00.1   1         Multimedia controller       Audio device                     NVIDIA Corporation

I have got a typo in my first post, I get on dmesg the error message that the "nvidia kernel modules (drm, uvm, modset etc) are not found"

[smira]

From what I can tell (and you can do this research yourself to verify my points), even latest 570.x doesn't support your GPU https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-570-148-08/index.html

It's Blackwell, but seems that RTX 5xxx is supported only with Ada Lovelace. So you might need to figure out first what is the NVIDIA supported driver in stable series (or is there going to be one?).

[flashpixx]

I have seen this article https://medium.com/@ttio2tech_28094/how-to-install-rtx-5090-5080-5070-ti-5070-drivers-on-linux-detailed-guide-d7069c7a0db7 for supporting the GPU, but I'm not quite sure that your link reference the correct hardware, because it is for data center GPU, but maybe I could be wrong

[flashpixx]

The driver seems to be the version 575.57.08
https://www.nvidia.com/en-us/drivers/details/245537/

[smira]

That is not a production version yet within NVIDIA, only 570.x is the production release. So I guess once backported it might be available, but also keep in mind, this is more consumer GPU, not a server GPU, so NVIDIA might support it different way.

[flashpixx]

Sure, but will this support by Talos itself, if the driver is available?
I think there are a lot of options to use not a server GPU, in general it would be nice if Talos supports both

[smira]

We ship two version of drivers that are marked by NVIDIA as Production and LTS. At the moment it is 570.x and 535.x.

[flashpixx]

So I have to wait for the 575 update, is this correct?

[smira]

Please see my response above. Sorry, we don't have any insight into the way NVIDIA supports your particular GPU in their drivers series, neither on the topic of production releases/versions.