stable sync elevated CPU usage

[romange]

to reproduce, run on server with replica and without:

./dfly_bench --qps -100 --pipeline 150 -test_time 300 --ratio=1:0 -d 32 --proactor_threads=2 -c 5

Also attaching pprof files (can be shown with pprof -http localhost:8080 profile_norepl.pb.gz)
It clearly shows that MainLoop grows from 7% to 31%.

profile_norepl.pb.gz

profile_withrepl.pb.gz

[adiholden]

We suspect that master sends lots of small packets to replica in stable sync, meaning that we don't aggregate a lot of journal changes before sending to master, but instead send small amount of journal changes in each packet. We should verify this is the case and if so try to understand what is the reason for it and how to solve this.
@mkaruza already did some local testing with master replica and dfly bench while using wireshark and saw small packets. I suggest @abhijat try to run the setup locally your self and if need any help with wireshark ping Mario.

[romange]

Note, it's not urgent but important to attend at some point.

Maybe first step would be to reproduce it.

[romange]

My setup was: master/replica on r6g.xlarge, dfly_bench on another machine.