A question on circuit relay

[dariusc93]

In a couple of projects I am using relay and dcutr but what I been doing is connecting to multiple circuits. My question would be is there any downsides to doing this? While I havent had any issues so far, I've been wondering to myself if i should avoid listening on multiple but instead listen on one, which is what I originally have done before deciding on this.

Whats pretty much being done

for relay_addr in relay_addrs {
        swarm.dial(relay_addr.clone())?;
        let mut sent = false;
        let mut recv = false;
        let mut delay = futures_timer::Delay::new(std::time::Duration::from_secs(20)).fuse();

        loop {
            futures::select! {
                event = swarm.select_next_some() => {
                    match event {
                        SwarmEvent::Behaviour(ChatBehaviourEvent::Identify(IdentifyEvent::Sent { .. })) => {
                            sent = true;
                        }
                        SwarmEvent::Behaviour(ChatBehaviourEvent::Identify(IdentifyEvent::Received { .. })) => {
                            recv = true;
                        },
                        _ => continue
                    }
                }
                _ = delay => {
                    break;
                }
            }
            if sent && recv {
                break;
            }
        }

        if !sent || !recv {
            continue;
        }
        let relay_listener = relay_addr.with(Protocol::P2pCircuit);
        swarm.listen_on(relay_listener)?;

    }

Although I cannot imagine it being performant long term I have not had any issues but wanted to know the best solution in this case. I know doing something like autorelay that would be more preferred, which is something I do plan on doing in the near future.

[thomaseizinger]

I can't think of any downsides. Listening on multiple relays will improve availability in case one of them is down.

One note regarding your loop: You could optimize this by dialling all relays in parallel and looking at the PeerId in the each identify event.

[dariusc93]

Thanks for the response and your suggestion. The snippet of code was mostly an mix example from the hole punching example and whats being used in a code i used to test hole punching, but i would likely change that to your suggestion

Also I do believe it would improve availability also, which is the reasoning why i thought of doing it this way, but cannot imagine listening on 50 of them (wouldnt do that much either way) but thought i get some insight on this since i dont think its documented anywhere about this scenario and implication of doing so with multiple.

[thomaseizinger]

Really it depends on how generous your relays are :)

The idea of relay-v2 is that relays are reasonably cheap to operate by being able to restrict bandwidth and general usage of their services. That makes it easier to have multiple relays which is better for the / a network.

[dariusc93]

I see. Although for the most part I am using it to connect with other peers with dcutr so i dont see it using much resources afterwards.

[thomaseizinger]

Yeah, that is the best case scenario really :)

[dariusc93]

Thank you. I think I do have a bit more info to proceed to test a few things out soon

[mxinden]

For what it is worth @dariusc93 here are the default values for the go-lib2p AutoRelay implementation, i.e. the component managing the connections to different relays:

https://github.com/libp2p/go-libp2p/blob/031ec50a04507cb2def993ef0ad15cf95eb98922/p2p/host/autorelay/options.go#L37-L45

Long term it would be wonderful for rust-libp2p to have a AutoRelay implementation as well. @dariusc93 let us know in case you would be interested in contributing that.

[dariusc93]

Thank you for your response @mxinden and sorry for the late reply. The link you sent is helpful and give me a bit more of an idea of what I could do in terms of managing the connections. I do want to experiment a little first but in the future I may just do that if nobody else do it by then.

[dariusc93]

Not sure if i should open up a new discussion or put the question here but i will probably go on and put my questions here and if I need to, i can open a different discussion, but I written an attempt in a autorelay. More of a PoC (so im bound to rewrite it anyway since this is implementation is a bit over the place while attempting to figure out the correct course). I collect a set of relays found after bootstrapping (pretty much checking the protocol and storing them in a behaviour) and use the one with the lowest ping to try to establish a reservation (probably a better determining factor but decided on that for now). however there are some questions I have:

1. Although later today I will be doing a deeper check with the logs, when a reservation is establish I am unable to establish a connection with the node when attempting to connect. Would there be any reason for this? I am only checking Event::ReservationReqAccepted and Event::ReservationReqFailed but should i be looking at any other event after a reservation is established?
2. Some relays reservation fails with Multistream select failed. Is there a deeper explaination to this? (again later i will be looking deeper at the logs to see whats going on so i might be able to provide something later on)
3. Sometimes I would get a resource limit exceeded when connecting externally using ipfs client (probably should use libp2p for testing) but nothing indicated this from the relay client event. What might be the best way to handle this on the client side?
   4.The main delay in another attempt was determining if im understanding the libp2p specs for circuit relay v2, but am I suppose to be checking for STOP or HOP when getting the Identify event? The spec shows The 'hop' protocol is client-initiated, and is used when clients send commands to relays; it is used for reserving resources in the relay and opening a switched connection to a peer through the relay. and The 'stop' protocol governs the endpoints of circuit switched connections., which makes me believe STOP would be what I look, but checking the protocols when relay v2 client is only used, that is what is listed. Checking for HOP instead.
4. There are times when I would see a something along the lines of /ip4/147.75.61.171/tcp/4001/p2p/12D3KooWQgmve2ZyEUWkLDJQPDk51fLh7NrxN3wMBhxrGAeKKs26/p2p-circuit/p2p/12D3KooWD53eviVdfDp6yCV6KwUpRMvTBRZw848wMmVebPWwJVcS/p2p/12D3KooWD53eviVdfDp6yCV6KwUpRMvTBRZw848wMmVebPWwJVcS but from what I see, nowhere in my code am I adding an additional p2p protocol to the multiaddr. Is this normal? I've noticed this even around the time of the initial question above but never put to much thought into it at the time.
5. (more of a side question) Will there ever be any plans to allow one to be able to listen on an address from a behaviour (eg using NetworkBehaviourAction when sending the event and return it when the behaviour is polled)? The current workaround I did was to send an event to the swarm and listen there instead once it been received.

EDIT:

I did read around the specs abit more and also inspected go-libp2p so it looks like I should be looking for HOP and not STOP.

I also noticed in go-libp2p that one way i could find relays by leveraging the kad providers so I may make changes to do that if how im doing it is not preferable

EDIT 2:

Right now question 3 think i have an idea for resolving it (though any opinion sand advice is appreciated). I think the only issue that might be more concerning regarding this is question 2 (will provide the logs soon) and 5 (though for 5 i might need to scan over the relay crate to see if its being done there). Is this intended to have multiple p2p protocols in the multiaddr when utilizing a relay or is it harmless? cc @mxinden

[mxinden]

Although later today I will be doing a deeper check with the logs, when a reservation is establish I am unable to establish a connection with the node when attempting to connect. Would there be any reason for this? I am only checking Event::ReservationReqAccepted and Event::ReservationReqFailed but should i be looking at any other event after a reservation is established?

That should be enough.

Some relays reservation fails with Multistream select failed. Is there a deeper explaination to this? (again later i will be looking deeper at the logs to see whats going on so i might be able to provide something later on)

I would need more information to make an informed guess.

Sometimes I would get a resource limit exceeded when connecting externally using ipfs client (probably should use libp2p for testing) but nothing indicated this from the relay client event. What might be the best way to handle this on the client side?

Where is the limit being exceeded? On the rust-libp2p node or on the go ipfs client?

There are times when I would see a something along the lines of /ip4/147.75.61.171/tcp/4001/p2p/12D3KooWQgmve2ZyEUWkLDJQPDk51fLh7NrxN3wMBhxrGAeKKs26/p2p-circuit/p2p/12D3KooWD53eviVdfDp6yCV6KwUpRMvTBRZw848wMmVebPWwJVcS/p2p/12D3KooWD53eviVdfDp6yCV6KwUpRMvTBRZw848wMmVebPWwJVcS but from what I see, nowhere in my code am I adding an additional p2p protocol to the multiaddr. Is this normal? I've noticed this even around the time of the initial question above but never put to much thought into it at the time.

This should be fixed with #2701, right?

(more of a side question) Will there ever be any plans to allow one to be able to listen on an address from a behaviour (eg using NetworkBehaviourAction when sending the event and return it when the behaviour is polled)? The current workaround I did was to send an event to the swarm and listen there instead once it been received.

I would not be opposed to a NetworkBehaviourAction::ListenOn. In general, I think the NetworkBehaviour capabilities should be a superset of the Swarm API surface.

I also noticed in go-libp2p that one way i could find relays by leveraging the kad providers so I may make changes to do that if how im doing it is not preferable

Yes. I think that is a reasonable approach. Namely to lookup e.g. ones own PeerId and make reservations with the 20 closest nodes. That said, I think your current approach is a viable solution, though might be prone to eclipse attacks.

Right now question 3 think i have an idea for resolving it (though any opinion sand advice is appreciated). I think the only issue that might be more concerning regarding this is question 2 (will provide the logs soon) and 5 (though for 5 i might need to scan over the relay crate to see if its being done there). Is this intended to have multiple p2p protocols in the multiaddr when utilizing a relay or is it harmless? cc @mxinden

See above.

Thanks @dariusc93 for putting in all this work! In case you think some synchronous time would be helpful, we could try to accommodate the timing of one of our rust-libp2p maintainer calls.

[dariusc93]

I would need more information to make an informed guess.

I will try to get you more info this weekend (if not this week) but from what I've seen is that the majority of them fail because of it the nodes being used only have Circuit V1relay. Unable to filter them out directly in the behaviour (awaiting #2680 to possibly aid in that). I could attempt to filter them out by cross checking with identify (especially if I use kad to discover the relays where i cant simply check the protocols of the peers when getting providers) before storing or making a reservation, just that might be a bit tricky with the way i want to handle it.

Where is the limit being exceeded? On the rust-libp2p node or on the go ipfs client?

With kubo. I am going try using rust-libp2p more (and possibly go-libp2p for comparison) though for testing, but with some relays I am having success while others I am either hitting a resource limit (not testing with dcutr at the moment)

This should be fixed with #2701, right?

I'd looked into this and it might be an issue with go-libp2p (or kubo since thats what I used). Using libp2p-lookup (as well as checking the addresses from Identify), i dont see the double p2p protocol while with go-ipfs, i do see it. Will have to look closely though.

I would not be opposed to a NetworkBehaviourAction::ListenOn. In general, I think the NetworkBehaviour capabilities should be a superset of the Swarm API surface.

I did open #3291 for it. Probably would wait for additional changes to be done.

Yes. I think that is a reasonable approach. Namely to lookup e.g. ones own PeerId and make reservations with the 20 closest nodes. That said, I think your current approach is a viable solution, though might be prone to eclipse attacks.

I will keep that in mind. In the client example, it does bootstrap kad but I do have a flag to lookup a random peerid as well as using kad providers.

Thank you again for the response.