etcd over kubespan?

[justyns]

I've been using tailscale (with the tailscale extension) to handle most of the internal traffic in my talos cluster. I'm working on testing a new cluster without tailscale. I have some confusion about kubespan though.

For testing, I have 5 servers. 3 in dc1, 2 in dc2. They all have public ipv4 and ipv6. The 3 in dc1 are the control plane servers, but do not have a separate private network.

I thought that etcd traffic would go through kubespan, but it looks like etcd still listens on the public ip instead. I wasn't able to get the cluster to bootstrap properly until opening up the ingress firewall rules to allow etcd (2379-2380) traffic. I also had to allow the trustd port (50001).

Should (or can) etcd go over the kubespan network by default?

Is there a list of recommended firewall rules when using kubespan? I saw the recommended rules on https://www.talos.dev/v1.10/talos-guides/network/ingress-firewall/ but it doesn't really mention what needs changed when using kubespan, if anything. I thought I'd be able to get rid of most of the firewall rules other than one for allowing 51820/udp, but that doesn't seem to be the case so far.

[smira]

There is not enough information to help you, but you have some issue which your configuration.

KubeSpan transparently and consistently captures and converts node-to-node traffic, so as long as it works, you don't need to do anything. KubeSpan traffic (inside the tunnel) is automatically whitelisted by the Ingress Firewall. KubeSpan ports should be of course whitelisted in the Ingress Firewall (UDP/51820).

[justyns]

I may have been misunderstanding how kubespan works.. I was focusing on the fact that etcd still showed as listening on the public ip, instead of the kubespan ipv6/ula address.

If kubespan is enabled (and working), does that mean traffic from nodeA's public ip to nodeB's public ip would automatically route through the kubespan wireguard tunnel?

[smira]

If kubespan is enabled (and working), does that mean traffic from nodeA's public ip to nodeB's public ip would automatically route through the kubespan wireguard tunnel?

yes, and Ingress Firewall whitelists KubeSpan traffic automatically (it should), so if you block public access, KubeSpan access should still work (if it doesn't, open an issue for it).