one to many secondary IP for a floating VIP with SG

[dstoy53]

Hey folks,

I'm experimenting with an advanced zone with security groups enabled in a shared network, and I'm trying to accommodate a keepalived VIP.

The traffic to the VIP only succeeds if it's assigned as a secondary IP to the active VM so that it gets added to iptables + ebtables. I've validated this behavior by failing over and then manually adding the VIP to the ip set and to the relevant ebtables chain. Trying to add the same secondary IP to the second VM via ui/api fails.

Without looking at everything else this idea might break, I think the easiest solution would be to allow a secondary IP to be attached to multiple VMs so the unmodified security group script can handle the rules.

Am I missing something obvious that could more easily solve this use case?

I also ran a small test that appeared to succeed:

1. for the first VM, add a secondary IP to act as the VIP
2. for the second VM, clone the relevant row in nic_secondary_ips replacing the vmId and nicId with those belonging to the 2nd VM (and satisfying the uuid constraint)
3. for the second VM, in the nics table toggle secondary_ip to 1
4. perform some live migrations to trigger security_groups.py on the target hosts

The VIP appears to fail over as expected when I start/stop keepalived in the VMs to trigger failover. I also checked iptables/ebtables and everything looks good there with the primary/secondary IPs for both VMs.

The UI shows the same secondary IP in each VM's NIC settings. The address list in Guest Networks shows the IP as allocated, though if I refresh the page it cycles between showing instance 01 vs 02.

Previous discussions on this topic tend to suggest modifying security_groups.py but I think this might provide a cleaner answer unless it breaks something else.

[weizhouapache]

it looks like a very good workaround.
we could add a flag (e.g. vip) to secondary ip . this kind of IP can be associated to multiple vms under same account.

just curious, anyone has used kubernetes clusters or containers inside vms on shared networks with security groups ?

[dstoy53]

I agree that some type of flag would be good for safety, otherwise an api user won't know if they're allocating a shared secondary IP without first checking the allocation status. The UI would already recognize the IP as allocated and hide it on the NIC page, so that might need some adjustments to enable this behavior.

I think kubernetes clusters with a cni managed overlay will work well enough since outbound traffic would get a SNAT with the primary VM IP of each node. In this scenario the inbound traffic would need a load balancer with a VIP for the control plane nodes and a VIP for an ingress controller (or other nodeport services), so only the VIPs would need to be able to float.

Attaching pods directly to the network would get more complicated and the CNI would need to talk to the cloudstack api to attach/detach IPs. This is what cilium does in aws. In AWS there's also prefix delegation, because if the CNI keeps requesting /32s for each pod you hit a maximum on the ENI, and if it keeps adding ENIs you hit the maximum ENIs for the instance type. Prefix delegation allocates a /28 to the ENI (consuming the same capacity as a single /32 on the ENI) and pod addresses are assigned from that prefix.

[weizhouapache]

I agree that some type of flag would be good for safety, otherwise an api user won't know if they're allocating a shared secondary IP without first checking the allocation status. The UI would already recognize the IP as allocated and hide it on the NIC page, so that might need some adjustments to enable this behavior.

I think kubernetes clusters with a cni managed overlay will work well enough since outbound traffic would get a SNAT with the primary VM IP of each node. In this scenario the inbound traffic would need a load balancer with a VIP for the control plane nodes and a VIP for an ingress controller (or other nodeport services), so only the VIPs would need to be able to float.

Attaching pods directly to the network would get more complicated and the CNI would need to talk to the cloudstack api to attach/detach IPs. This is what cilium does in aws. In AWS there's also prefix delegation, because if the CNI keeps requesting /32s for each pod you hit a maximum on the ENI, and if it keeps adding ENIs you hit the maximum ENIs for the instance type. Prefix delegation allocates a /28 to the ENI (consuming the same capacity as a single /32 on the ENI) and pod addresses are assigned from that prefix.

thanks for the sharing @dstoy53

Allocating /28 subnet to the ENI seems a good idea.
I do not know how security groups works on the IPs (maybe allow all traffic on hypervisor level and let ingress controller to manage ?)

[rajujith]

Good workaround @dstoy53