load balancing: slow coordinator avoidance

[kbr-]

It would be nice to have a real load balancing strategy which chooses coordinators based on actual metrics such as how fast it can process requests.

See scylladb/scylladb#5715 for motivation. I explain in the comments what the issue is (it is a driver-side issue).

The 4.x versions of the Java driver AFAIK do have such strategies - at least one.
From https://docs.datastax.com/en/developer/java-driver/4.11/manual/core/load_balancing/#built-in-policies:

DefaultLoadBalancingPolicy should almost always be used; it requires a local datacenter to be specified either programmatically when creating the session, or via the configuration (see below). It can also use a highly efficient slow replica avoidance mechanism, which is by default enabled.

[piodul]

Just leaving a note: support for latency-aware load balancing policy has been merged, however the new Java driver's DefaultLoadBalancingPolicy uses a different algorithm. I hope that the Java driver's algorithm performs better - Java driver deprecated the latency aware policy and later introduced the default one - so we should implement something similar and compare how well it performs relative to latency aware policy.