Question about fetch.wait.max.ms and fetch.min.bytes default values.

[dbrachet]

Hello,

We are facing some significant CPU usage increase on kafka broker when increasing the number of consumers (different groups but same topic for testing purpose) connected to the broker even if no data is published to the topic. We have tested this using kafkacat/kcat utility (based on librdkafka) and we have found that increasing the value of fetch.wait.max.ms in our kafka customers significantly reduce the pressure on the kafka broker.

I am now trying to understand why default values for fetch.wait.max.ms and and fetch.min.bytes are respectively 500 ms and 1 byte for a kafka consumer and what would be the impact of increasing fetch.wait.max.ms. Since the default value of fetch.min.bytes is 1 it seems to me that increasing fetch.wait.max.ms will not have any impact on consumer latency and throughput since data will be delivered "immediately" (when poll is done) to the consumer. Is there any reason why fetch.wait.max.ms is not higher by default (increasing fetch.min.bytes does not reduce cpu usage of kafka broker in idle mode)? What would be the impact in customer if I increase etch.wait.max.ms but keep fetch.min.bytes to 1?

Thanks

[edenhill]

The main reason for fetch.wait.max.ms not being set too high by default is avoiding to block the connection for too long in case there are no new messages.
But you should be okay with increasing the waiting time.

[dbrachet]

Thanks a lot, it makes a lot of sense.