Suboptimal P99 latency under pipelining workloads

[romange]

We have not really focused on combination of P99 latency and pipelining, especially in the context
of uncoordinated omission access patterns.

When investigating the elevated latency phenomena even under small CPU load I noticed the following:

1. The connection fiber yields after reading 11 requests from the socket. This limits how much optimization we can do further down the road when processing batches of requests.
2. When reading from the socket we limit the socket buffer to 4KB, unless a single request requires more. Which means that when reading series of small requests from the socket we can read at most K request that fit into 4KB. This causes a similar effect of the pipelining efficiency.
3. For cluster mode, we shard our request by tags. Valkey Cluster spec allows processing a multi-key command that touches multiple tags that belong to the same slot if, i.e. mget {foo}aaa {bar}aaa is allowed as long as both {foo} and {bar} belong to the same shard. Dragonfly processes such requests correctly but this means these commands run on multiple shards. Unfortunately pipelining optimizations do not work well with multi-shard commands and loose this efficiency (can be fixed in the future)

[romange]

Regarding (1): we should still yield in the connnection fiber to let the AsyncFiber to unload requests - otherwise we will keep reading from the socket until all the data is read or we reached the pipelining limit. It is sub-optimal because of the lost opportunity to kick off the pipeline running in parallel to reading from the socket.

There are two possible solutions:

1. Simple: To add a configurable counter limit and yield like we do today (default of 11 is still probably too low).
2. Add a more sophisticated self tuning heuristic. For that we could take into account:
   a. Hardcoded uper limit (1000?)
   b. dispatch queue memory usage: after a certain threshold - yield
   c. How many commands we could squash in one step: there is no point adding much more than this - if we will end up dividing the dispatch stream into a smaller batches for squashing.

[romange]

Finally whatever we do - we should comment why we do it and even link this issue for more details.

[romange]

Here is the general overview of the flow + the bugs we found