Skip to content

Metadata request doesn't handle topics length correctly #5049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ProBrian opened this issue Apr 24, 2025 · 0 comments · May be fixed by #5053
Open

Metadata request doesn't handle topics length correctly #5049

ProBrian opened this issue Apr 24, 2025 · 0 comments · May be fixed by #5053

Comments

@ProBrian
Copy link

ProBrian commented Apr 24, 2025

Context

I'm using kcat as kafka client, when running below command in shell
kcat -b 127.0.0.1:9092 -L -d metadata -X allow.auto.create.topics=true
the debug output shows that client will generate two metadata requests:

%7|1745481050.866|METADATA|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Request metadata for brokers only: connected
%7|1745481050.866|METADATA|rdkafka#producer-1| [thrd:app]: 127.0.0.1:9092/bootstrap: Request metadata for all topics: application requested
%7|1745481050.867|METADATA|rdkafka#producer-1| [thrd:main]: 127.0.0.1:9092/bootstrap: ===== Received metadata: connected =====
%7|1745481050.867|METADATA|rdkafka#producer-1| [thrd:main]: 127.0.0.1:9092/bootstrap: ClusterId: 5L6g3nShT-eMCtK--X86sw, ControllerId: 1
%7|1745481050.867|METADATA|rdkafka#producer-1| [thrd:main]: 127.0.0.1:9092/bootstrap: 1 brokers, 0 topics
%7|1745481050.867|METADATA|rdkafka#producer-1| [thrd:main]: 127.0.0.1:9092/bootstrap: Broker #0/1: localhost:9092 NodeId 1
%7|1745481050.868|METADATA|rdkafka#producer-1| [thrd:main]: 127.0.0.1:9092/bootstrap: ===== Received metadata: application requested =====
%7|1745481050.868|METADATA|rdkafka#producer-1| [thrd:main]: 127.0.0.1:9092/bootstrap: ClusterId: 5L6g3nShT-eMCtK--X86sw, ControllerId: 1
%7|1745481050.868|METADATA|rdkafka#producer-1| [thrd:main]: 127.0.0.1:9092/bootstrap: 1 brokers, 2 topics
%7|1745481050.868|METADATA|rdkafka#producer-1| [thrd:main]: 127.0.0.1:9092/bootstrap: Broker #0/1: localhost:9092 NodeId 1
%7|1745481050.868|METADATA|rdkafka#producer-1| [thrd:main]: 127.0.0.1:9092/bootstrap: Topic test-topic with 1 partitions
%7|1745481050.868|METADATA|rdkafka#producer-1| [thrd:main]: 127.0.0.1:9092/bootstrap: Topic __consumer_offsets with 50 partitions
%7|1745481050.868|METADATA|rdkafka#producer-1| [thrd:main]: 127.0.0.1:9092/bootstrap: Caching full metadata with 1 broker(s) and 2 topic(s): application requested

I captured packets with wireshark and check those two meta requests, and found the second metadata request for all topics has an incorrect topics list length, below is the capture screenshot,
the first one:

Image

the second one:

Image

As we can see the second metadata request use a int32 to describe the topics length(0x 00 00 00 00), while the first metadata request use the correct 1-based uvarint(0x 01).

I checked the source code, and see the difference may be caused by https://github.com/confluentinc/librdkafka/blob/master/src/rdkafka_request.c#L2803-L2826
for the case topic_cnt == 0, we don't do varint encoding.
It seems for the same API version(Metadata Request Version 12), there's a mismatch behavior of encoding.
I think for the topic_cnt == 0 && ApiVersion >= 9 case, we should also invoke rd_kafka_buf_finalize_arraycnt to do varint encoding.

ProBrian added a commit to ProBrian/librdkafka that referenced this issue Apr 25, 2025
…fluentinc#5049)

For metadata request API version >= 9, Null array length is not correctly
set, the 4 bytes int should be replaced by 1 byte varint.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant