StatefulSet pod FQDN DNS resolution fails with containerd

[fmunteanu]

Environmental Info:

K3s Version:

# k3s -v
k3s version v1.32.6+k3s1 (eb603acd)
go version go1.23.10

Node(s) CPU architecture, OS, and Version:

# uname -a
Linux apollo 6.8.0-1030-raspi #34-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 23 05:50:33 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration:

# kubectl get nodes
NAME     STATUS   ROLES                       AGE   VERSION
apollo   Ready    control-plane,etcd,master   34m   v1.32.6+k3s1
boreas   Ready    control-plane,etcd,master   31m   v1.32.6+k3s1
cerus    Ready    control-plane,etcd,master   31m   v1.32.6+k3s1
chaos    Ready    <none>                      29m   v1.32.6+k3s1
crios    Ready    <none>                      29m   v1.32.6+k3s1
helios   Ready    <none>                      29m   v1.32.6+k3s1
hermes   Ready    <none>                      29m   v1.32.6+k3s1
hypnos   Ready    <none>                      29m   v1.32.6+k3s1

Server configuration:

# cat /etc/rancher/k3s/config.yaml
data-dir: /var/lib/rancher/k3s
bind-address: 192.168.4.2
cluster-domain: cluster.local
disable:
  - coredns
  - local-storage
  - metrics-server
  - servicelb
  - traefik
disable-cloud-controller: true
disable-kube-proxy: true
disable-network-policy: true
embedded-registry: true
etcd-arg:
  - --election-timeout=5000
  - --heartbeat-interval=500
etcd-expose-metrics: true
flannel-backend: none
kube-controller-manager-arg: bind-address=192.168.4.2
kube-scheduler-arg: bind-address=192.168.4.2
kubelet-arg:
  - config=/etc/rancher/k3s/kubelet.yaml
node-taint:
  - node.cilium.io/agent-not-ready:NoExecute
  - node-role.kubernetes.io/control-plane:NoSchedule
server: https://192.168.4.10:6443
tls-san:
  - 192.168.4.10
token: redacted

Agent configuration:

# cat /etc/rancher/k3s/config.yaml
data-dir: /var/lib/rancher/k3s
kubelet-arg:
  - config=/etc/rancher/k3s/kubelet.yaml
node-taint:
  - node.cilium.io/agent-not-ready:NoExecute
server: https://192.168.4.10:6443
token: redacted

Describe the bug:

StatefulSet pods cannot resolve each other's FQDNs, breaking clustering applications like AlertManager. I installed VictoriaMetrics and I get DNS resolution fails:

Failed to resolve vmalertmanager-vmks-1.vmalertmanager-vmks.kube-system.svc.cluster.local.:9094: 
lookup vmalertmanager-vmks-1.vmalertmanager-vmks.kube-system.svc.cluster.local. on 10.43.0.10:53: no such host

Expected behavior:

AlertManager pods should resolve peer FQDNs like vmalertmanager-vmks-1.vmalertmanager-vmks.kube-system.svc.cluster.local for cluster communication, subdomain doesn't appear in alertmanager pods automatically, as per K8s design.

See VictoriaMetrics/helm-charts#2283 for troubleshooting details.

[brandond]

There is nothing that we customize in this project that would alter this behavior and I have no reason to suspect this is an defect in K3s.

You've not shared any information about the actual deployment you're working with here, but you should read https://kubernetes.io/docs/concepts/services-networking/dns-pod-service and ensure that what you're deploying meets the requirements for creation of DNS records you're expecting. Note that it specifically requires presence of a service with name matching the pod's subdomain value.

Also, you've disabled coredns, so I have no idea whether or not whatever you've replaced it with (assuming you've replaced it with something) is behaving properly.

Given you appear to have disabled pretty much everything bundled with k3s (all the packaged components, several kubernetes components, and the cni) I'm not really sure what you still expect us to be responsible for.

[fmunteanu]

@brandond thank you for looking into this. The cluster runs error-free with ArgoCD, cert-manager, Cilium, CoreDNS, external-dns, kured, Longhorn, metrics-server, VictoriaLogs and VictoriaMetrics helm charts deployed independently. The only hiccup is related to the issue detailed above. CoreDNS follows the K3s standards, here are the custom helm chart settings:

hpa:
  enabled: true
  maxReplicas: 3
  minReplicas: 1
  metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
podDisruptionBudget:
  maxUnavailable: 1
resources:
  limits:
    memory: 128Mi
  requests:
    cpu: 10m
    memory: 128Mi
rollingUpdate:
  maxSurge: 1
  maxUnavailable: 0
servers:
  - zones:
      - zone: .
    port: 53
    plugins:
      - name: errors
      - name: health
        configBlock: lameduck 5s
      - name: ready
      - name: kubernetes
        parameters: cluster.local in-addr.arpa ip6.arpa
        configBlock: |-
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
      - name: prometheus
        parameters: 0.0.0.0:9153
      - name: forward
        parameters: . /etc/resolv.conf
      - name: cache
        parameters: 30
      - name: loop
      - name: reload
      - name: loadbalance
service:
  clusterIP: 10.43.0.10
serviceAccount:
  create: true

There is an old #1834 issue that was never resolved, hence why I allowed myself to open another one. Please let me know your thoughts, you have a lot of experience with K3s. Thank you again!

Edit: DNS records work properly, example:

[brandond]

You'd need to share the pod (deployment/replicaset/statefulset) yaml, along with the yaml of service that you believe should be triggering creation of the DNS records. I'm not going to deploy the whole chart (and your custom coredns deployment to boot) to test it for you.

[fmunteanu]

@brandond, please see VictoriaMetrics/helm-charts#2283 (comment) for Pod (where we deal with missing subdomain, as detailed into #1834 issue), StatefulSet and Service yaml sources. I don't see a Deployment and ReplicaSet for alertmanager, I believe that is because VM use Prometheus into their operator. If I missed something, please let me know so I can provide the info to you.

Edit: The issue makes surface when I use a cluster of 2 alertmanager pods, required to fix the status warning.

[brandond]

As demonstrated at VictoriaMetrics/helm-charts#2283 (comment) - this does not reproduce on a stock K3s cluster. You've misconfigured something in your extensive customization.