Is there a way to increase log level of Talos services (e.g., containerd)?

[zombiemaker]

I would like to increase the logging level for Talos services to help troubleshoot configuration issues. Any way of doing this?

[zombiemaker]

I tried using the machine configuration parameter debug set to true, but I see no significant increase in log messages.

Some background:

• my Talos nodes are having issues pulling the container image for kubelet that is hosted in a private container image registry (distribution registry) on the LAN
• getting the following error message in dmesg: " level=info msg=fetch failed error=failed to do request: Head "https://my-container-registry-host/v2/kubernetes/kubelet/manifests/v1.32.3": dial tcp 192.168.1.99:443: i/o timeout host=my-container-registry-host image=my-container-registry-host/kubernetes/kubelet:v1.32.3"
• the system is able to pull images from docker.io when using the "talosctl image pull" command
• I would like to get more visibility into what exactly is the problem by increasing log detail level

[zombiemaker]

Thank you for your response!

Assuming that you were referring to the container registry host, I am able to push and pull images from the container registry using Docker CLI.

[rothgar]

But it doesn't look like you're able to pull images from the node. That's why I suggest looking at the network traffic on the node.

[frezbo]

you could also launch a node shell via kubectl -n kube-system debug -it --profile=sysadmin node/<node-name> --image=alpine and do an apk install crane && crane pull <your-image> /tmp/image and see if that works, then if it works drop the --profile=sysadmin and check again, that should rule out networking issues

[zombiemaker]

@rothgar - I took your suggestion on packet capturing from the Talos node using the talosctl tcpdump command. I did packet captures using the BPF filter "ip host {ip-address-of-container-registry}". That data showed that the node transmitted packets out but did not receive any response. I am working through the process of setting up and capturing packets from the switch. I will post the results when I have them.

@frezbo : Thanks for your input! The Talos node currently does not have kubelet running (it is unable to pull the container image for kubelet). So I cannot perform any Kubernetes-level operations at the moment.

[zombiemaker]

@rothgar : I was able to use talosctl tcpdump along with Wireshark to sniff / capture packets transmitted through the switches. I had some VLAN configuration and routing issues. Thank for your suggestion.

It still would be nice to have the ability to increase log levels for the different Talos services (e.g. machined, containerd, etc.).