[Producer] rd_kafka_conf_set_stats_cb() json object with negative offset values

[hari269it]

Hi,

We are using librdkafka C library.

The producer code is setting up the rd_kafka_conf_set_stats_cb() for receiving the stats and dumping them for debugging.
It is observed that few fields in partitions have default values -1001 (RD_KAFKA_OFFSET_INVALID)

Fields with default vaule of -1001 are eof_offset, query_offset, commited_offset, lo_offset, hi_offset, consumer_lag, stored_offset and consumer_lag having -1
However, other stats from the topics are incrementing and non negative.

Queries:

1. Are these fields eof_offset, query_offset, commited_offset, lo_offset, hi_offset, consumer_lag, stored_offset applicable to both producer and consumer or only consumer?
2. In case the above fields are applicable for producer stats as well, what configurations/api calls need to be called for correct stats updates of the fields.

Sample log:

"partitions": {
"0": {
"partition": 0,
"leader": 1,
"desired": false,
"unknown": false,
"msgq_cnt": 0,
"msgq_bytes": 0,
"xmit_msgq_cnt": 0,
"xmit_msgq_bytes": 0,
"fetchq_cnt": 0,
"fetchq_size": 0,
"fetch_state": "none",
"query_offset": -1001,
"next_offset": 0,
"app_offset": -1001,
"stored_offset": -1001,
"commited_offset": -1001,
"committed_offset": -1001,
"eof_offset": -1001,
"lo_offset": -1001,
"hi_offset": -1001,
"consumer_lag": -1,
"txmsgs": 6701688,
"txbytes": 20214656430,
"rxmsgs": 0,
"rxbytes": 0,
"msgs": 6701688,
"rx_ver_drops": 0,
"msgs_inflight": 0,
"next_ack_seq": 0,
"next_err_seq": 0,
"acked_msgid": 0
},

Searched for similar discussions and issues, but all are related to consumer and suggests to call rebalance ( which is present in our consumer code).
Appreciate if the queries can be clarified.

Thanks

[edenhill]

-1001 is, as you've discovered, the default/unset offset value in librdkafka. So those metrics can safely be ignored until they have a non -1001 value.
Some consumer-specific metrics are unfortunately included in producer stats, and vice versa, due to the way metrics generation is implemented.
We could perhaps update https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md with an indication if it is a producer or consumer metric.

[hari269it]

Thanks a lot for the response.