forked from tylerholien/milena
-
Notifications
You must be signed in to change notification settings - Fork 5
Open
Description
The function offsetCommit works fine for storing the offset, but fails to store the corresponding metadata (you can verify this using offsetFetch
)
Digging into the code, I found two reasons that this fails....
- According to the binary protocol spec (https://kafka.apache.org/protocol), the definition of
OffsetCommitRequest
is wrong:
The definition in milena is
newtype
OffsetCommitRequest = OffsetCommitReq (ConsumerGroup, [(TopicName, [(Partition, Offset, Time, Metadata)])]) deriving (Show, Eq, Serializable, Generic)`
however, the Time
field should not be there for apiVersion=0
(you can verify that the metadata is not getting through by running kafka verbose.... the metadata that is actually read is incorrect in the logs)
- More importantly, even after the but in item 1 is fixed (by removing the
Time
field), no the metadata is still not saved. I am not 100% certain, but it looks like Kafka has deprecated offsetCommit metadata in apiVersion=0 (It looks like this is so from reading the code)
Note that apiVersion=0 was supposed to store offsets and metadata in zookeeper, apiVersion=1 moved to storing in a kafka topic.
We have gotten metadata working by moving to higher apiVersion for offsetCommit/offsetFetch, after I get things cleaned up, I can provide a PR with the fix
Metadata
Metadata
Assignees
Labels
No labels