Cannot Pull Container Image For kubelet Hosted On Private Container Registry

[zombiemaker]

Background:

• running Talos v1.9.5
• Talos nodes are having issues pulling the container image for kubelet that is hosted in a private container image registry on the LAN
• receiving the following error message in talosctl dmesg: " level=info msg=fetch failed error=failed to do request: Head "https://distribution-registry.local-domain/v2/kubernetes/kubelet/manifests/v1.32.3": dial tcp ip-address-of-container-registry:443: i/o timeout host=distribution-registry.local-domain image=distribution-registry.local-domain/kubernetes/kubelet:v1.32.3"
• the Talos node is able to pull images (e.g., nginx:latest) from docker.io when using the "talosctl image pull" command
• the Talos node is NOT able to pull images from private container registry when using the "talosctl image pull" command
• the container registry software is CNCF Distribution Registry
• no authentication is needed to access container images
• the container registry server application uses TLS x509 certificate signed by a private CA
• the certificates for the private root and intermediate CAs are added to the Talos machine config using the TrustedRootsConfig YAML document
• the following machine config excerpt is used for defining the container registry

registries:
  mirrors:
    distribution-registry.local-domain
      endpoints:
        - https://distribution-registry.local-domain/
      skipFallback: true
      overridePath: false
  config:
    distribution-registry.local-domain:
      # tested with and tls.insecureSkipVerify: true   

• the Talos machines are connected to a switch that uses MAC-based VLAN association
• the Talos machine configurations do not have configuration for VLANs
• IP connectivity between the Talos machines and the LAN DNS server and container registry has been checked from the container registry node and the switch
• the containerd configuration files in the Talos nodes have been checked and seem fine
  • /etc/containerd/config.toml
  • /etc/cri/conf.d/cri.toml
  • /etc/cri/conf.d/hosts/my-container-registry-host/hosts.toml

Anyone has any suggestions on where the problem is? What else should I be checking?

Thanks in advance!

[smira]

You have the error message which tells exactly the reason - you don't have network connectivity to your registry, whether it's firewall, MTU, misconfigured routing, or anything else is impossible to guess.

But it's not TLS yet, and it's not any configuration related to the registry itself.

[zombiemaker]

Agreed. In another discussion thread, I am working through network packet capturing at the switch to see where the problem is.

Thank you!