Upgrade to 3.5.18/17 does not handle the same load handled by 3.5.9

[suryaprakashm1]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

We have been using etcd 3.5.9 successfully, but after upgrading to 3.5.17/3.5.18, we noticed that it struggles with the same load, with range queries taking over 10 minutes. We are working to reproduce this issue outside of production. How can I capture all incoming requests to etcd for analysis?

What did you expect to happen?

3.5.17/3.5.18 should not face any problem in handling the load, which was working fine with 3.5.9

How can we reproduce it (as minimally and precisely as possible)?

Do you have any suggestions for capturing all incoming request logs?

I've enabled debug logging for etcd, but currently, the logs only show range queries that are taking a long time. I don't see all incoming requests being logged.

Anything else we need to know?

We're running a 3-member etcd cluster handling approximately 14,000+ range queries and 4,000 transaction queries per second.

Etcd version (please run commands below)

$ etcd --version
# paste output here

$ etcdctl version
# paste output here

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

[serathius]

We have been using etcd 3.5.9 successfully, but after upgrading to 3.5.17/3.5.18

Only thing we can do is search through changes between those two versions and try to pinpoint once that could have affected you. Range 3.5.9->3.5.17 is large gap, could you maybe help narrowing it down? Maybe downgrade to 3.5.13 and see if that helps. That would help us bisect the source of the issue.

[serathius]

/cc @ahrtr

[serathius]

Possibly it's the #19406

[serathius]

It affected v3.5.16-v3.5.18 and was fixed in v3.5.19

[ahrtr]

First, please try the latest patch 3.5.22, and let's know whether you still see the same performance problem. If yes (still have the problem), then as @serathius mentioned above, can you please help us to narrow down the issue of scope?