Skip to content

offsetCommit doesn't properly store metadata #11

@jamshidh

Description

@jamshidh

The function offsetCommit works fine for storing the offset, but fails to store the corresponding metadata (you can verify this using offsetFetch)

Digging into the code, I found two reasons that this fails....

  1. According to the binary protocol spec (https://kafka.apache.org/protocol), the definition of OffsetCommitRequest is wrong:
    The definition in milena is
    newtype OffsetCommitRequest = OffsetCommitReq (ConsumerGroup, [(TopicName, [(Partition, Offset, Time, Metadata)])]) deriving (Show, Eq, Serializable, Generic)`

however, the Time field should not be there for apiVersion=0
(you can verify that the metadata is not getting through by running kafka verbose.... the metadata that is actually read is incorrect in the logs)

  1. More importantly, even after the but in item 1 is fixed (by removing the Time field), no the metadata is still not saved. I am not 100% certain, but it looks like Kafka has deprecated offsetCommit metadata in apiVersion=0 (It looks like this is so from reading the code)

Note that apiVersion=0 was supposed to store offsets and metadata in zookeeper, apiVersion=1 moved to storing in a kafka topic.

We have gotten metadata working by moving to higher apiVersion for offsetCommit/offsetFetch, after I get things cleaned up, I can provide a PR with the fix

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions