Load-balancing for iroh endpoints

[St4NNi]

I need to manually configure additional direct addresses for an iroh endpoint beyond what the automatic discovery mechanisms provide. My setup has a load balancer with dedicated ports for iroh endpoints that are not direct interfaces on the host but are direct addresses that other peers should use to connect.

Is there a way to manually add these load balancer addresses to an endpoint after creation, or should this be configured during the endpoint builder phase? Are there any considerations for how manually set addresses interact with the automatic discovery mechanisms?

The automatic address detection only sees the internal host interfaces, but I need peers to connect via the load balancer's external addresses and dedicated ports.

[flub]

Let me check if I understand you correctly:

• You have a server which can be reached on a certain socket address which is managed by a load-balancer.
  • This address is not the address of a local interface returned by the OS.
  • This address is not discovered using QAD (previously STUN - it does the same thing).
    • That means an outgoing connection from the iroh endpoint does not go through this network path.
• Clients should send to this load-balanced socket address.

And you're asking how to tell the server's iroh endpoint about this load-balanced socket address?

IIUC you have a load-balancer that binds a UDP socket. Traffic it receives is split between a number of backend servers and blindly forwarded to them. Probably based on client's ip + port?

But what happens to the return traffic? Do you expect the backend servers to send directly from their own socket and you don't route those packets through the load-balancer? So clients would receive packets from a different ip + port than they send to?

[St4NNi]

First of all thanks for the fruitful discussion. I guess I should change this discussion from the Q&A category to General or even Ideas.

For anyone else following along:

TL;DR:

It is in the current version of iroh not possible to manually set direct addresses.

---

Assuming the load-balancer addresses are publicly reachable and do not require holepunching, can you create a NodeAddr containing just those addresses in? That is, no RelayUrl, no other direct addresses. If you can distribute those NodeAddrs to your clients, do the connections just work, via the load-balancer and without holepunching?

Yes, I just tried this, and it works. It creates a direct connection without any issues. The only issue is that the server node will not see the "real" IP address of my client, only the one from the load balancer.

Another interesting question is how the load-balancer copes with client migration? If the client gets a NAT rebinding and suddenly starts sending UDP packets from a different source address will it make sure to send them to the same backend server?

Again, yes this is a very interesting question unfortunately I don't really know how to test this out. I would need a client that switches its source address, maybe a phone ?!

The QUIC solution to this is that the load balancer to supposed to look at the Connection ID (CID) in each packet and if it is an existing CID it can send it to the backend server that's handling that CID, which will then detect the client migration and make sure the connection stays up. But a naive NodePort service would not do this I guess? So your connections would break.

Just to clarify, our setup is not just a single NAT from the load-balancer but also an additional NAT from the kube-proxy / NodePort. So the network route is more like:

  graph LR;
      LoadBalancer-->K8sNode;
      K8sNode-->|via NodePort Service| kube-proxy;
      kube-proxy-->iroh-pod

Loading

Both the load balancer and the kube proxy will perform NAT because the load balancer forwards packets to any Kubernetes node, which may not be the node on which the iroh pod is running. Since we only run a single iroh instance, shouldn't this iroh instance also receive the CID and handle it ? Again, i would love to try this out in some way.

If you did use a relay connection however that path would stay alive and the connection would survive and establish a direct connection to the backend server again (but not via the load-balancer, just like before).

Yes understand.

Another QUIC-level way of handling this is to always create a new path directly to the backend server after the connection is established via the load-balancer (thinking in unreleased iroh 1.0 terms now, which will use QUIC multipath). Especially if that doesn't need holepunching. I think in iroh this might always happen anyway, because iroh the server will always tell the client about alternative addresses it can be reached on. The client will then try those paths and check the latency and choose the path with the better latency, which hopefully is the path directly to the backend server. Now if the client migrates the backend server would handle it natively without cooperation of the load balancer. But this relies on the connection always migrating to the backend server directly. (And of course does not allow for keeping connections alive across backend server restarts, but that's anyway not going to happen with a NodePort load-balancer.)

This sounds interesting, too. Does it also work the other way around? Does iroh take into consideration the path the client took to reach the server? The server could use this information to automatically detect the additional address. For example, if the client informs the server that it connected via address X, the server would be aware of this alternative path and could add it to the list of direct addresses.

[flub]

Another interesting question is how the load-balancer copes with client migration? If the client gets a NAT rebinding and suddenly starts sending UDP packets from a different source address will it make sure to send them to the same backend server?

Again, yes this is a very interesting question unfortunately I don't really know how to test this out. I would need a client that switches its source address, maybe a phone ?!

My usual strategy is to use my laptop, connect it via a phone-provided wifi hotspot first and then switch it to the home wifi. That way it's super easy to get code to run on the device that moves network, which is a bit harder to do on a phone directly.

But yes, these days phones switching between mobile and wifi are the most common occurrences of devices hot-switching networks. Followed by laptops suspending and resuming on a different wifi.

[flub]

Another QUIC-level way of handling this is to always create a new path directly to the backend server after the connection is established via the load-balancer (thinking in unreleased iroh 1.0 terms now, which will use QUIC multipath). Especially if that doesn't need holepunching. I think in iroh this might always happen anyway, because iroh the server will always tell the client about alternative addresses it can be reached on. The client will then try those paths and check the latency and choose the path with the better latency, which hopefully is the path directly to the backend server. Now if the client migrates the backend server would handle it natively without cooperation of the load balancer. But this relies on the connection always migrating to the backend server directly. (And of course does not allow for keeping connections alive across backend server restarts, but that's anyway not going to happen with a NodePort load-balancer.)

This sounds interesting, too. Does it also work the other way around? Does iroh take into consideration the path the client took to reach the server? The server could use this information to automatically detect the additional address. For example, if the client informs the server that it connected via address X, the server would be aware of this alternative path and could add it to the list of direct addresses.

No, the client never does so. If it did it would not be information the server could generally trust, so it would only apply for this one connection. A client might be lying to mess with the server.

I think the best solution is still what you suggested at the start: allow applications to add other sources of "direct/public addresses" to an endpoint.

[flub]

First of all thanks for the fruitful discussion. I guess I should change this discussion from the Q&A category to General or even Ideas.

Thanks for the discussion as well! We've always known we'll have to deal with load-balancers some day but have postponed doing so thus far.

Assuming the load-balancer addresses are publicly reachable and do not require holepunching, can you create a NodeAddr containing just those addresses in? That is, no RelayUrl, no other direct addresses. If you can distribute those NodeAddrs to your clients, do the connections just work, via the load-balancer and without holepunching?

Yes, I just tried this, and it works. It creates a direct connection without any issues. The only issue is that the server node will not see the "real" IP address of my client, only the one from the load balancer.

That's not a big deal I think? iroh anyway doesn't really operate on IP addresses but rather on NodeIds.

I think this is the most promising solution for you short term. Though you lose the relay server backup path which is not ideal and will break clients that move between networks.

Both the load balancer and the kube proxy will perform NAT because the load balancer forwards packets to any Kubernetes node, which may not be the node on which the iroh pod is running. Since we only run a single iroh instance, shouldn't this iroh instance also receive the CID and handle it ? Again, i would love to try this out in some way.

You'll need to have a load-balancer that knows about QUIC for this, because it needs to know about the CIDs. All of those (think internally inside cloudflare, google, facebook - I'm not aware of public ones) coordinate CIDs between the backend servers and the load balancers. That is the load balancer will be aware of which CIDs are issued by which backend server, usually by signing or encrypting some data and using that as the CID (the CID is an arbirary-length sequence of opaque bytes).

Simply observing CIDs is not enough since they do change during the lifetime of a connection. You can of course hope they don't change around the same time as client migrations happen, though hope is not a strategy 😉.

Though writing this, you can use a custom ConnectionIdGenerator that never rotates CIDs on your server. Though RFC9000 will frown upon you 😉.

[All8Up]

I was just working through a bit of this myself and I suspect my use case is only partially covered by this discussion. There are two items I'm trying to solve: redundancy and balancing. A key difference is that I will not allow clients to directly connect into the backend, they will always be forced through a relay because the backend is a private secure cloud fully firewalled and internally subdivided among different subnets. At the end of the day, Iroh fits great for my needs behind the scenes, it is only the initial hop from client to gateway which seems to be a sticky point for me.

The following is an outline I put together for the worst case scenario in terms that it has not been resolved by the time I need to start implementing such things. Keep in mind that I'm a bit out of date, haven't worked on an infrastructure solution in about 10 years so take this with a grain of salt... I'm basically just proving to myself that I can probably make it work if I have to. I don't think it would be a terrible hacky solution but I'm also not in the guts of Iroh (yet):

• Modify Iroh DNS and endpoint lookup to support a redirect list.

  • Likely just have an invalid endpoint as the primary result and a set of attached records.
    • Each record represents one of the redundant gateways into the private cluster. They have a short TTL (time to live) and the gateway will update it regularly to handle liveliness.
    • I have not dug into the code, I am 'assuming' the standard DNS abilities exist... Famous last words...

• The initial lookup in the core of Iroh will probably need to be modified to support the DNS redirect.

  • Leave it as defaulted to off in the endpoint builder.
  • Use the dns records to find an actual end point. Records could include NA West, NA East, EU etc for geo help. ?
  • Possibly a reconnect flag which will re-query the DNS, not sure this is required, haven't looked at the code and I think this was a point of Quic to make reconnect fast so ... another end point flag perhaps.

• Create the 'gateway' out of the Iroh Relay server.

  • I can't think of a better starting point since I need to relay things from the public NIC to the private NIC anyway.
  • The gateway will not support any ALPNs natively, it will require backend services to connect to the private NIC to tell the gateway what they support and then the gateway can relay to the lowest load for the specific ALPN.
  • Add heartbeats to the ALPN suppliers for failover needs and kick the clients to anything that dies back to reconnection.

All said and done, that is where I am in thinking the problem out. I have a few other issues I'll have to cover but this is the related bits to load balancing that I've considered.