Next-hop selection for multihomed cluster nodes

[andrenth]

Hello

I have a Kubernetes cluster running with Calico where the control plane nodes are configured as route reflectors which establish BGP sessions with the worker nodes and publishes load balancer IP routes to a router outside of the K8S cluster. This is working as expected, as the router receives routes to the load balancer IPs /32 setting the next-hop to the IP address of the worker nodes where the service is scheduled.

However, all my nodes have two network interfaces (public and private), and the above is working fine over the public network. Now I'm trying to host internal services in this cluster, having them bound to load balancer IPs from the private network.

I've added a router to the private network and it is able to establish BGP sessions with the control plane nodes. However, the issue I'm facing is that it receives routes to the private load balancer IPs with the next-hop still set to public IP address of the worker nodes, because the next-hop is set to the autodetected node IP.

I'd like to be able to be able to select the next-hop so that I get routes like this:

public-service/32 via public-worker-ip
private-service/32 via private-worker-ip

Since this doesn't seem to be possible, does anyone know of a workaround?

[caseydavenport]

Hm, the BGPPeer resources does have a sourceAddress option, but I am not sure either of them will quite get you what you want here:

Specifies whether and how to configure a source address for the peerings generated by this BGPPeer resource. Default value "UseNodeIP" means to configure the node IP as the source address. "None" means not to configure a source address.

https://docs.tigera.io/calico/latest/reference/resources/bgppeer

[andrenth]

It would be nice if the sourceAddress option could accept an ip address string, or maybe an interface name. I’m not familiar with the code though, not sure what the complexity to implement that would be.