How to avoid connection limits preventing desired behavior?

[nathanielc]

I am experiencing a behavior in my network where connection limits prevents two peers from connecting when my protocol needs them to connect. Concretely I am trying to share some data around the network using bitswap + dht (kademlia). I am consistently hitting an issue where a small number of nodes (one or two) provide the needed data and the peer that wants the data is not already connected to the providing peers. Bitswap does its thing doesn't find the data and then queries the DHT at which point it learns about the providers but when it attempts to connect to those peers and ask for the data the initial connection is blocked because of the max_outgoing_connections limit. At this point bitswap essentially hangs forever because it cannot connect to the peers and the data is never retrieved.

If I force nodes to be initially connected to fewer peers such that the limit is not reached everything works as expected. However peers tend to want to connect and the network seems to stabilize at the limit. Meaning peers will greedily connect to more peers until their limits are reached and then stop. This is generally good behavior because a more connected network is better.

How do I reconcile both of these behaviors on the network?

Is there a way to have priority connections such that a new outgoing connection would be allowed when a protocol requests it?

Is there a way to set the limits in such a way that the stable behavior is not to saturate the limits allowing for some overhead in more temporary connections?

Is there some other concept I am missing in this context?

Thanks

[dariusc93]

At the moment, there is no actual connection manager in rust-libp2p that you may find in go-libp2p or js-libp2p where the node may shed or disconnect less helpful peers when it comes to new ones. I've been hitting a similar situation and been experimenting on a small connection manager where it would tag useful peers (eg being used in a protocol like gossipsub, bitswap, part of the routing table in kad, etc) and any that arent tagged (not used anyplace other than there establishing a connection) would be disconnected on a set interval or on any pending incoming or outgoing connection if it hits or near limit, with a list of peers who may be whitelisted from this limit. It works to some extend but honestly could be better, but the recent changes done to the behaviour in regards to connections made it easier to try out.

I guess in your situation, you could write a behaviour that combines the connection limit behaviour but also perform a check of less than helpful peers and disconnect them before performing any connections with the peers found over DHT with them having higher priority. Eg maybe any old peers you exchanged blocks with via bitswap but no longer need you could disconnect from them and once the disconnection is confirmed you can connect to the new peers.

[nathanielc]

Thanks this is helpful. I can see a similar strategy working for my use case as well. Tracking which peers are participating in protocols I care most about and then disconnecting from peers that are not really helping as much.

Do you have your behavior anywhere I could take inspiration from? I am currently stuck on a version of libp2p that doesn't have the recent connection changes but looks like this will help motivate upgrading.