Implement BGP

[sebiklamar]

Hi Vegard,

as I saw you having BGP support on your roadmap list, I want to share my documentation and experience of my BGP w/ Opnsense including EgressGateway PoC.

What's in the pocket

1. IN: Ingress (IngressGateway) with any IP you want, i.e. you don't need to have a matching interface for that. Just name an IP to your liking in the io.cilium/lb-ipam-ips annotation as usually and cilium will add it to the BGP routing.
2. IN: L2 routing/reachability of your BGP-managed IPs because your default gateway will do a RA for your hosts still residing in the same L2 net (your 192.168.1.0/24 subnet), i.e. you can switch of your L2Announcement code. I didn't further investigate this b/c this is not a relevant scenario in my environment (k8s nodes in a separate server VLAN).
3. IN w/ workaround: Egress (EgressGateway) with BGP iff you have an Ingress defined in parallel. Due to the limitation mentioned in the next (4.) item there's no BGP advertisement for the Egress. Hence, you need to define an Ingress in parallel (counter-measure 1) and you need to ensure the Egress is set up for the same node as the pod using the Ingress is running (counter-measure 2) , i.e. you will need to pin (nodeSelector) both the Deployment and the CiliumEgressGatewayPolicy.
   NB: For Ingress-only scenario (1.) no pinning is needed at all, i.e. you will have full flexibility.
4. OUT: Egress-only scenario (without any Ingress for the same IP). Would need Cilium Enterprise Ed., cf. Cilium blog post for Cilium Enterprise 1.15.
5. INCOMPATIBLE: If you want to use EgressGateway feature, you would need to live without CiliumEndpointSlice feature (cf. cilium issue 24833). CiliumEndpointSlice is currently enabled in your environment.

How-to

TLDR

For seeing all pieces in action see in my evolving homelab repo based on your setup

• openssh app for EgressGateway (and IngressGateway) example.
• additional-values.yaml file for cilium installation, cf. details below.
• the cilium base and envs/dev folders contain the Resource definitions for BGP

Links

• Opnsense BGP: https://docs.opnsense.org/manual/dynamic_routing.html#bgp-section
• Cilium BGP: https://docs.cilium.io/en/stable/network/bgp-control-plane/bgp-control-plane-v2/#bgp-control-plane-v2
• outdated (using BGPPeeringPolicy), though good opnsense docu: https://baremetalblog.com/posts/tech/2024-03-12-cilium-bgp-and-you/
• EgressGateway: https://docs.cilium.io/en/stable/network/egress-gateway/egress-gateway/

opnsense guide

See the 3rd link listed above which is having good screenshot documentation on opnsense setup.

My documented install notes from Obsidian:

1. Install os-frr package (from Plugins tab)
2. Open firewall port tcp/179 on interface K8S
3. Configure Peers [[#Configuration]]

Warning message during install

Beware that remote control of frr8 daemons over TCP sockets is enabled by
default.
Use daemon flags in /etc/rc.conf to disable it if unneeded, for example:
zebra_flags="-P0"
ospfd_flags="-P0"

FRR's OSPF daemons tries to allocate big socket buffer, so generate warning
messages like:
"setsockopt_so_sendbuf: fd 6: SO_SNDBUF set to 1048576 (requested 8388608)"
To prevent such message kern.ipc.maxsockbuf can be increased:
sysctl kern.ipc.maxsockbuf=16777216

Configuration

Routing > General

Field	Value
Enabled	true
Enable CARP Failover	true
Enable SNMP AgentX Support	default: false
Enable Logging	default: true
Log Level	default: Notifications
Firewall rules	default: true

Routing > BGP

General

Field	Value
Enabled	true
BGP AS Number	64520
Network
Log Neighbour Changes	default: false
Route Redistribution	Connected routes (directly attached subnet or host)

Neigbours

Field	Value
Enabled	true
Description	dev-work-01 / -02
Peer-IP	10.7.8.134 / .135
Remote AD	64523
Update Source Interface	K8S
Next-Hop-Self	Enabled
all other fields

k8s guide

Ensure BGP is enabled in cilium

# additional-values.yaml
bgpControlPlane:
  enabled: true

For also having Egress feature available, cilium values.yaml file needs to be at least:

# additional-values.yaml
## enable BGP
# URL: https://docs.cilium.io/en/stable/network/bgp-control-plane/bgp-control-plane/#installation
bgpControlPlane:
  enabled: true

## enable egress gateway
# URL: https://docs.cilium.io/en/stable/network/egress-gateway/egress-gateway/
egressGateway:
  enabled: true
bpf:
  masquerade: true

# egress gateway not compatible with CiliumEndpointSlice
# URL: https://github.com/cilium/cilium/issues/24833
enableCiliumEndpointSlice: false

FYI: I currently maintain a base values.yaml and an env.-specific additional-values.yaml file. Though, I didn't manage yet to maintain the values.yaml file only once in the base folder with only patches being applied in the environments as overlay.

Resource Definitions

Overview

• base/CiliumBGPAdvertisements: Defines prefixes that are injected into the BGP routing table.
• base/CiliumBGPPeerConfig: A common set of BGP peering setting. It can be used across multiple peers.
• <env>/CiliumBGPClusterConfig: Defines BGP instances and peer configurations that are applied to multiple nodes, specific to an <environment>.
• CiliumBGPNodeConfigOverride: Defines node-specific BGP configuration to provide a finer control, not used.

<env>/CiliumBGPClusterConfig

environment-specific

---
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp
spec:
  nodeSelector:
    # disable BGP on control plane nodes
    matchExpressions:
      - key: node-role.kubernetes.io/control-plane
        operator: DoesNotExist

  bgpInstances:
    - name: "instance-64523"
      localASN: 64523 # Use your cluster's ASN here!
      peers:
        - name: "peer-64520-gw2"
          peerASN: 64520
          peerAddress: 10.7.8.2
          peerConfigRef:
            name: "cilium-peer"
        - name: "peer-64520-gw3"
          peerASN: 64520
          peerAddress: 10.7.8.3
          peerConfigRef:
            name: "cilium-peer"

ASNs

Just pick an ASN ID from the private AS number range 64.512 – 65.534 (16b) or from the newer 32b range 4.200.000.000 - 4.294.967.294.

Most documented setups (incl. mine at the moment) use different ASNs for the router (e.g. opnsense) and partner (k8s cluster), although, AFAIK, the same ASNs can be used (it's the same party/system "Vegard's system/environment").

Cheat Sheet

cilium bgp peers
cilium bgp routes

Have fun with a modern routing setup with certainly more robust connectivity (instead of the k8sClientRateLimit hack for L2), and true source IP.

HTH -- Sebastian

Edit: Added cilium folders links in TLDR section as openssh only covers Egress and not BGP and links.

[vehagn]

@sebiklamar Thanks for the info! This is a goldmine once I start testing it out.

I've been way too busy these last months, but I think I'll be able to look at BGP with Cilium and UniFi soon™

Other resources I have for this are

• https://baremetalblog.com/posts/tech/2024-03-12-cilium-bgp-and-you/
• https://www.packetswitch.co.uk/bgp/
• https://medium.com/@spudstr/ubiquity-unifi-k3s-bgp-and-metallb-744b50706c7c
• https://docs.cilium.io/en/latest/network/bgp-control-plane/bgp-control-plane-v2/