Unable to get nebula system extension setup in 1.10.2

[admun]

I am trying to bring up a Talos node with nebula system extension setup, but not able to. I can use some help to debug the issue.

Env: Fedora 42, talos running from libvirt/KVM VM, booted with XPE.

Setup procedure:

1. Create a XPE boot image, see here
2. download the vmlinux and initramfs-amd64
3. In Fedora, setup a virtual network with matchbox and a iXPE script as follow

#!ipxe
kernel http://192.168.x.x:8080/assets/talos/vmlinuz-amd64 talos.platform=metal initrd=initramfs.xz init_on_alloc=1 slab_nomerge pti=on console=tty0 printk.devkmsg=on ${cmdline}
initrd http://192.168.x.x:8080/assets/talos/initramfs-amd64.xz
boot

4. Then create a VM and boot a talos node into maintenance mode via XPE
5. Create talos config with talosctl gen config with a patch.yaml that disable CNI and proxy
6. run talosctl apply-config and then talosctl bootstrap
7. run talosctl kubeconfg and run helm to install cilium CNI

After the control-plane node is up, it seems that nebula service is not coming up, even with a ExtensionServiceConfig for it

Please see https://taloscommunity.slack.com/archives/CG25RPZNE/p1748196408367289 and let me know how I should proceed to debug further.

[frezbo]

Please attach a support bundle, and the extensionserviceconfig used without any secrets

[admun]

Where I can find the support bundle?

see here's the extensionserviceconfig

---
apiVersion: v1alpha1
kind: ExtensionServiceConfig
name: nebula
configFiles:
  - content: |
        pki:
          ca: /usr/local/etc/nebula/ca.crt
          cert: /usr/local/etc/nebula/node.crt
          key: /usr/local/etc/nebula/node.key
        static_host_map:
          "x.x.x.x": ["my-lighthous.somewhere.net:4242"]
        lighthouse:
          am_lighthouse: false
          interval: 30
          hosts:
            - "x.x.x.x"
        listen:
          host: 0.0.0.0
          port: 0
        punchy: true
        respond: true
        delay: 1s
        tun:
          dev: nebula1
          drop_local_broadcast: false
          drop_multicast: false
          tx_queue: 500
          mtu: 1300
          routes:
          unsafe_routes:
        logging:
          level: debug
          format: text
        firewall:
          conntrack:
            tcp_timeout: 120h
            udp_timeout: 3m
            default_timeout: 10m
            max_connections: 100000
        outbound:
          - port: any
            proto: any
            host: any
        inbound:
          - port: any
            proto: any
            host: any
    mountPath: /usr/local/etc/nebula/config.yml
  - content: |
        -----BEGIN NEBULA CERTIFICATE-----
        -----END NEBULA CERTIFICATE-----
    mountPath: /usr/local/etc/nebula/ca.crt
  - content: |
        -----BEGIN NEBULA CERTIFICATE-----
        -----END NEBULA CERTIFICATE-----
    mountPath: /usr/local/etc/nebula/node.crt
  - content: |
        -----BEGIN NEBULA ED25519 PRIVATE KEY-----
        -----END NEBULA ED25519 PRIVATE KEY-----
    mountPath: /usr/local/etc/nebula/node.key

[frezbo]

talosctl support

[admun]

Please see attached
support.zip

[frezbo]

you've an invalid extensionservice config

2025-05-28T04:42:16.542Z ESC[31mERRORESC[0m error loading extension service spec {"component": "controller-runtime", "controller": "runtime.ExtensionServiceController", "filename": "nebula.yaml", "error": "error unmarshalling extension service config: yaml: line 9: did not find expected key"}

[frezbo]

Also the dmesg is not complete, could you reboot and attach a support zip

[admun]

you've an invalid extensionservice config

2025-05-28T04:42:16.542Z ESC[31mERRORESC[0m error loading extension service spec {"component": "controller-runtime", "controller": "runtime.ExtensionServiceController", "filename": "nebula.yaml", "error": "error unmarshalling extension service config: yaml: line 9: did not find expected key"}

Thanks for spoting it. I will look into it.

Meanwhile, how do I spot this kind of error form system extension... only from this support bundle?

sorry that I might not have done enough rtfm

[admun]

I am not able to determine the extensionserviceconfig error.... line 9 is part of the nebula's own config file I think, and get extensionserviceconfig -o yaml looks have the expected contents

talosctl --talosconfig talosconfig -n $TALOS_1 get extensionserviceconfig -o yaml
node: 192.168.x.x
metadata:
    namespace: runtime
    type: ExtensionServiceConfigs.runtime.talos.dev
    id: nebula
    version: 1
    owner: runtime.ExtensionServiceConfigController
    phase: running
    created: 2025-05-28T21:21:45Z
    updated: 2025-05-28T21:21:45Z
spec:
    files:
        - content: |
            pki:
              ca: /usr/local/etc/nebula/ca.crt
              cert: /usr/local/etc/nebula/node.crt
              key: /usr/local/etc/nebula/node.key
            static_host_map:
              "192.168.x.x": ["my-lighthouse.somewhere.net:4242"]
            lighthouse:
              am_lighthouse: false
              interval: 30
              hosts:
                - "192.168.x.x"
            listen:
              host: 0.0.0.0
              port: 0
            punchy: true
            respond: true
            delay: 1s
            tun:
              dev: nebula1
              drop_local_broadcast: false
              drop_multicast: false
              tx_queue: 500
              mtu: 1300
              routes:
              unsafe_routes:
            logging:
              level: debug
              format: text
            firewall:
              conntrack:
                tcp_timeout: 120h
                udp_timeout: 3m
                default_timeout: 10m
                max_connections: 100000
              outbound:
                - port: any
                  proto: any
                  host: any
              inbound:
                - port: any
                  proto: any
                  host: any
          mountPath: /usr/local/etc/nebula/config.yml
        - content: |
            -----BEGIN NEBULA CERTIFICATE-----
            -----END NEBULA CERTIFICATE-----
          mountPath: /usr/local/etc/nebula/ca.crt
        - content: |
            -----BEGIN NEBULA CERTIFICATE-----
            -----END NEBULA CERTIFICATE-----
          mountPath: /usr/local/etc/nebula/node.crt
        - content: |
            -----BEGIN NEBULA ED25519 PRIVATE KEY-----
            -----END NEBULA ED25519 PRIVATE KEY-----
          mountPath: /usr/local/etc/nebula/node.key

[frezbo]

good point, let me look into this more

[admun]

please see the updated
support.zip

[frezbo]

Okay seems it's a bug in the extension definition itself:

Will see if we can get this fixed for 1.10.3, or else 1.10.4

[admun]

Thank you for looking into it.

I will put this aside for now and wait for a update

[admun]

just to report back that this issue has fixed in 1.10.3