Client disconnection after 5 minutes

[Yedidyah27]

Hey all,
We are experiencing constant disconnections using the admin client. Based on the following issue, #3703, the client is supposed to recover on its own. We are seeing the same disconnections as reported in the issue but it seems the client is unable to reconnect in some cases and we need to restart our application constantly.

[2022-03-24 07:28:17,156] Facility=FAIL, Level=Info, Message=[thrd:ssl://kafka1-0.test.net:9095/bootstrap]: ssl://kafka1-0.test.net:9095/0: Disconnected (after 299980ms in state UP, 1 identical error(s) suppressed), Name=rdkafka#producer-1
[2022-03-24 07:28:17,156] INFO  Kafka client log. Facility=ERROR, Level=Error, Message=[thrd:app]: rdkafka#producer-1: ssl://kafka1-0.test.net:9095/0: Disconnected (after 299980ms in state UP, 1 identical error(s) suppressed), Name=rdkafka#producer-1  
[2022-03-24 07:33:17,160] INFO  Kafka client log. Facility=FAIL, Level=Info, Message=[thrd:ssl://kafka1-0.test.net:9095/bootstrap]: ssl://kafka1-0.test.net:9095/0: Disconnected (after 299974ms in state UP, 1 identical error(s) suppressed), Name=rdkafka#producer-1 
[2022-03-24 07:33:17,160] INFO  Kafka client log. Facility=ERROR, Level=Error, Message=[thrd:app]: rdkafka#producer-1: ssl://kafka1-0.test.net:9095/0: Disconnected (after 299974ms in state UP, 1 identical error(s) suppressed), Name=rdkafka#producer-1  
[2022-03-24 07:38:17,166] INFO  Kafka client log. Facility=FAIL, Level=Info, Message=[thrd:ssl://kafka1-0.test.net:9095/bootstrap]: ssl://kafka1-0.test.net:9095/0: Disconnected (after 299987ms in state UP, 1 identical error(s) suppressed), Name=rdkafka#producer-1  
[2022-03-24 07:38:17,166] INFO  Kafka client log. Facility=ERROR, Level=Error, Message=[thrd:app]: rdkafka#producer-1: ssl://kafka1-0.test.net:9095/0: Disconnected (after 299987ms in state UP, 1 identical error(s) suppressed), Name=rdkafka#producer-1

I was wondering if the same client and port are used when reconnecting? Should we see logs if reconnecting fails? Maybe the reconnecting failures could be caused by port exhaustion in our system.

We are using librdkafka v1.8.2, Kafka brokers version 2.8 and network is configured to ssl.

[elibarel10]

Happened to me as well!

[edenhill]

The client will only log disconnects, you would need to enable debugging (debug=broker) get more detailed connection logging.

The client will always reconnect automatically. Failures to (re)connect will be logged.

Disconnects by themselves are typically not a problem, see https://github.com/edenhill/librdkafka/wiki/FAQ#why-am-i-seeing-receive-failed-disconnected

[Yedidyah27]

@edenhill thanks for your quick response. I do see that the kafka client is constantly disconnecting and then manages to reconnect in our logs. On the other hand, it looks like we are reconnecting with a different socket every time (see the logs). Since my application is running on a virtual machine, at some point I am guessing the sockets aren't released properly and we get port exhaustion.
Are all sockets properly released after disconnecting (ssl)?
Kafka logs are attached.

kafka_fail_logs.txt

[edenhill]

We don't have any reports of socket leakage and we do run our tests with socket/fd-tracking enabled.

If your process ran out of file-descriptors it would fail to create new (socket) connections, but that doesn't seem to be the case here.