filter messages by certain key/header without consuming, possible?

[tigerinus]

In our environment, we have a topic with 1000+ message flooding in per second, and 20+ micro-services written in Java and Python subscribed to the topic. Each service consumes only the messages it is interested depending on a particular message key and/or header.

In Java it seems to be able to use RecordFilterStrategy to filter messages without consuming each actual message.

Where as in Python, which depends on a librdkafka based Python binding, it consumes entire batch each time, then is able to fitler.

Question: Is prefiltering without consuming the messages also possible with librdkafka?

Thanks!

[edenhill]

Prefiltering is not supported (and I believe RecordFilterStrategy is a Spring thing, not Kafka).

You will need to filter out messages in your application logic.
If this is not performant enough (because Python), then consider making a Kafka Streams job that reads from your input topic and outputs only the filtered messages you want to an output topic that you then read from your Python application.