eth: add logic to drop peers randomly when saturated

[cskiraly]

WIP: this is still wip for a few reasons:

• conditions of dropping a peer should be improved, including also inbound peer drop.
• I've observed a deadlock in one test, still to be checked if the new module was the reason or not.

As of now, Geth disconnects peers only on protocol error or timeout, meaning once connection slots are filled, the peerset is largely fixed.

As mentioned in #31321, Geth should occasionally disconnect peers to ensure some churn. What/when to disconnect could depend on:

• the state of geth (e.g. sync or not)
• current number of peers
• peer level metrics

This PR adds a very slow churn using a random drop.

[fjl]

If you want to track the sync status of the node, this code will have to live outside of package p2p. It'll likely need to be created from eth.Ethereum instead.

[fjl]

let's rename type to dropper as well

[cskiraly]

Dropper in action:

• red lines are outbound drops, starting when outbound peer pool is saturated,
• yellow lines are inbound drops, starting much later, when inbound pool is saturated.
  This was with an accelerated drop timer. Default interval between drop events is set higher.

[fjl]

I think this is too high. At the current settings, it means each connection gets at least 14min of being peered before we consider it. I think it should be independent of the interval and a bit smaller, like 5min.

[fjl]

Or maybe 10min? Either way it's not intuitive for me why this should be connected to the interval.

[cskiraly]

Good point. The protection period is meant for peers to have time to start contribution, but at the moment we have no logic related to "start contribute", so the number is a bit arbitrary.

The logic in connecting the two intervals is from the other side of things: for not protecting a new node too long. What I wanted to avoid is protecting a node for too long, while the whole old peerset can be changed. Actually, for this it would be better to use peerDropIntervalMin ....

Let's make it independent for now, then we can consider again later. I would like this number to be bigger than the time a the dropped peer needs to find a replacement. 10min sounds safe to start with, then we can checked the effect when it is rolled out, and adjust.

[fjl]

Maybe best to remove this setting. I wouldn't want people to tinker with this, since a non-zero setting here can lead to weird behavior if peer finding efficiency is bad.

[cskiraly]

changing this will not change the drop interval, which ultimately drives churn in the system.

Changing this is somewhat similar to changing maxpeercount, which we do allow people to change (even on command line).
The effect is different on the dialed and on the served pool.

• On the dialed pool, it leads slightly more aggressive dial after drop, because dial slots are adaptive.
• On the served pool, it leads to more free slots in the system, which is largely needed, but needs careful checks before we turn this up.

So I would prefer keeping it to simplify experimenting with it for us.