Connection reset

[qiaojialin]

All versions:

[SteveYurongSu]

The scenarios that can trigger the issue often have hundreds or thousands of session connections, and the read/write load on these connections is high.

2021-07-19 09:00:56,431 [pool-8-IoTDB-RPC-Client-1026] ERROR o.a.t.s.TThreadPoolServer$WorkerProcess:258 - Thrift Error occurred during processing of message. 
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset

One root cause of the issue is that the default value for the system parameter SOMAXCONN is too low.

SOMAXCONN is /proc/sys/net/core/somaxconn default value.

It has been defined as 128 more than 20 years ago.

Since it caps the listen() backlog values, the very small value has
caused numerous problems over the years, and many people had
to raise it on their hosts after beeing hit by problems.

Google has been using 1024 for at least 15 years, and we increased
this to 4096 after TCP listener rework has been completed, more than
4 years ago. We got no complain of this change breaking any
legacy application.

For more detail: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=19f92a030ca6d772ab44b22ee6a01378a8cb32d4

[SteveYurongSu]

Way to resolve:

Set SOMAXCONN to a larger value
https://github.com/apache/iotdb/pull/3867/files