Talos 1.10 load nvidia extensions

[Svet-Izov]

I've tested Talos 1.9.5 on a physical machine and editing the controlplane.yaml, install, extensions I was able to load the GPU modules necessary for Talos to see the GPU. I've of course also applied the GPU patch (machine, kernel, modules, nvidia, uvm, drm and modeset) and that worked ok. On 1.10 however the extensions in controlplane.yaml seem to be deprecated (message was popping up on 1.9.5 as well) and I can't load the nvidia modules. No matter if I build the image with imager and include the following as part of the ISO:
ghcr.io/siderolabs/nonfree-kmod-nvidia-production:570.124.06-v1.10.0
docker.io/nvidia/k8s-device-plugin:1.10
nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
The nvidia modules are not loading and it must be some sort of a configuration error I'm doing.
Any ideas?
Thanks!

[frezbo]

Could you attach a support bundle via talosctl support? Also you'd need to perform an upgrade with the generated installer image for the extensions to take effect

[mapetersson]

im using as mentioned the imager docker way to build custom ISOs. How do i patch system extensionduring building the iso ? My docker way: cat profile.yaml | docker run --rm -i -v $PWD/_out:/out -v $PWD/image-cache.oci:/image-cache.oci:ro ghcr.io/siderolabs/imager:1.10.0 -

In my profile.yaml i have the system extension part added, is this sufficient or ?

Can i add flags (patch)to include system extensions during iso creation? Or do i need to build the iso and then apply patch after it in some way ?

[smira]

Please see documentation.

[marcopornic]

I have tried with this method (image-cache and system-extension-image at the same time):

docker run --rm -t \
  -v $PWD/_out:/secureboot:ro \
  -v $PWD/_out:/out \
  -v $PWD/image-cache.oci:/image-cache.oci:ro \
  -v /dev:/dev \
  --privileged \
  ghcr.io/siderolabs/imager:v1.9.5 \
  iso \
  --image-cache /image-cache.oci \
  --system-extension-image ghcr.io/siderolabs/iscsi-tools:v0.1.6@sha256:8ad7cd682f06198a6df7906d439939cc3f5feaa8a32f38ab2563268a17862baa \
  --system-extension-image ghcr.io/siderolabs/util-linux-tools:2.40.4@sha256:42e1d7685c92b66701b4c63d7daef37ac6b43822b3263dec29dd25b935532da0

Is it the correct way ? When I bootstrap my cluster I'hve no installed extension:

/talos# talosctl get extensions
NODE   NAMESPACE   TYPE   ID   VERSION   NAME   VERSION

Everything has to be deployed in air-gapped environment

[smira]

Please follow the documentation, the ISO is only used to boot the machine, the actual extensions installed in this case come from the installer image (but Image Cache comes from an ISO).

You can use Image Factory in the air-gapped environment as well as long as you make the image available (e.g. via a mirror).

[Svet-Izov]

At this point I think the logs will be irrelevant. It's the ISO we're creating that is causing the troubles. We'll read more and see how far we get.
Thanks for the advice above smira!

[Svet-Izov]

support.zip
Hello, uploading the support logs. We've build the ISO as per the imager instructions in the guide, including the system extensions we wanted to have, but we still see the nvidia module not found in the logs.
ISO built with the following:
docker run --rm -t -v $PWD/_out:/out ghcr.io/siderolabs/imager:v1.10.0 iso --system-extension-image ghcr.io/siderolabs/nvidia-container-toolkit-production:570.124.06-v1.17.5@sha256:35f307bd087f19297ff5f59fec10813ee1fa3bfbc479990521e00ea5bde70772 --system-extension-image ghcr.io/siderolabs/nonfree-kmod-nvidia-production:570.124.06-v1.10.0@sha256:945368ab333c72ff09f1189c9a297dd1e59e1c795f4754212642fbbfdd13a89a --extra-kernel-arg net.ifnames=0 --extra-kernel-arg=-console --extra-kernel-arg=console=ttyS1
gpu patch yaml:
machine: kernel: modules: - name: nvidia - name: nvidia_uvm - name: nvidia_drm - name: nvidia_modeset sysctls: net.core.bpf_jit_harden: 1

The modules I see loaded are:
iTCO_wdt 12288 0 - Live 0xffffffffc04df000
iTCO_vendor_support 12288 1 iTCO_wdt, Live 0xffffffffc04d5000
ahci 45056 1 - Live 0xffffffffc04be000
watchdog 40960 1 iTCO_wdt, Live 0xffffffffc03fd000
i40e 614400 0 - Live 0xffffffffc0409000
libie 8192 1 i40e, Live 0x0000000000000000
nvme 57344 0 - Live 0xffffffffc03dc000
lpc_ich 28672 0 - Live 0xffffffffc03cb000
i2c_i801 36864 0 - Live 0xffffffffc03b8000
libahci 49152 1 ahci, Live 0xffffffffc03a3000
mfd_core 12288 1 lpc_ich, Live 0xffffffffc039a000
i2c_smbus 12288 1 i2c_i801, Live 0xffffffffc0390000

Thanks in advance for your help!

[marcopornic]

Yes I have a Jenkins connected to internet which:

• generate talos ISO (image cache with extentions)
• generate some zarf packages to install Longhorn, Traefik, metallb, ...

Then I have a disconnected HyperV server where VM are created (booted from Talos ISO) and a disconnected "bastion" VM (ubuntu) where I execute my talosctl and zarf commands.

[mapetersson]

Image factory is awesome and for personal use & homelab env i would not hesitate to use it due to its simplicity and convenience. Its a different thingy for a company since you need to secure https://factory.talos.dev/ . This is the reason why our aim is to not to rely on https://factory.talos.dev/. Lets see how far we get since we are kind of stuck :( . If we dont manage to solve this then we will go for your hybrid approach @marcopornic :) . Hosting our own image factory is a complex beast and using OMNI will be a long discussion to convince finance to give us the credit card :D

[Svet-Izov]

@marcopornic Your air-gapped example is good and obviously works, but I'm trying to create an ISO with an installer image, created locally, that has all the extensions I need.
Let me show you what I mean:

Download the nvidia images:
crane export ghcr.io/siderolabs/nvidia-container-toolkit-production:570.124.06-v1.17.5 nvidia-toolkit.tar
crane export ghcr.io/siderolabs/nonfree-kmod-nvidia-production:570.124.06-v1.10.0 nvidia-kmod.tar

Edit profile.yaml to add the following:

systemExtensions:
- tarballPath: /nvidia-toolkit.tar
- tarballPath: /nvidia-kmod.tar
output:
kind: installer
outFormat: raw

That produces the installer.tar with the nvidia images inside.
I then build the installer with:

cat profile.yaml | docker run --rm -i
-v $PWD/_out:/out
-v $PWD/nvidia-toolkit.tar:/nvidia-toolkit.tar
-v $PWD/nvidia-kmod.tar:/nvidia-kmod.tar
ghcr.io/siderolabs/imager:v1.10.0 -

Now I need another yaml to create the ISO with the installer inside (createISO.yaml):

version: v1alpha1
architecture: amd64
output:
path: /out/talos-metal.iso
format: iso
imageCache:
installer:
path: /installer.tar

I've also tried with:

The createISO.yaml content is:

arch: amd64
platform: metal
secureboot: false
version: v1.10.0
input:
kernel:
path: /usr/install/amd64/vmlinuz
initramfs:
path: /usr/install/amd64/initramfs.xz
baseInstaller:
imageRef: installer:custom
imageCache:
imageRef: ""
ociPath: /image-cache.oci
output:
kind: iso
outFormat: raw

And then I try to create the ISO with:

docker run --rm
-v $PWD/_out:/out
-v $PWD/installer.tar:/installer.tar
-v $PWD/createISO.yaml:/createISO.yaml
ghcr.io/siderolabs/imager:v1.10.0
build --profile /createISO.yaml

Referencing the createISO.yaml with cat as in the installer example above didn't work for me. Neither did this.

My problem is I don't know how to reference the installer.tar so that I can build the ISO. It might be a stupid thing but I can't sort it out.
Using the image factory is plane simple, but I don't want to be dependent on it, so I'm looking for ways to build it locally.
Do you have any ideas on what I might be doing wrong?

Thanks!

[marcopornic]

Hello @Svet-Izov

To be honest, I'm not entirely sure why it's not working in your setup, but maybe you could try using the --extra /installer.tar flag with imager build? It might help the builder locate your local installer tarball properly.

Maybe @smira can help for this ?

If you end up finding a solution, I’d really appreciate it if you could share it here!

[smira]

I think at this point this discussion is very hard to follow, unformatted data (please use Markdown), so I can't parse anything.

If there's a specific question, please open a new discussion, make sure it's readable and reproducible.