Improve vector kafka source performance

[LinTechSo]

Hi all,
i want to ask if you could help or provide a working configuration for improving Kafka consuming performance with vector dev Kafka source.
i want to know how can i parallel Kafka consuming or etc.
my current configuration:

        type: kafka
        bootstrap_servers: kafka.default.svc:9092
        commit_interval_ms: 2000
        fetch_wait_max_ms: 1000
        session_timeout_ms: 20000
        socket_timeout_ms: 60000
        group_id: test
        topics:
          - test_topic

but my problem is my Kafka lag is too high per topic
for example the rate of Kafka produce on specific topic is about 1800 rps and the vector consume with 10 replica is 1200 rps (k8s env)

could you help me figure this out . i think there should be a configuration to increase Kafka librdkafka parallelism within vector or something

using vector version 0.25.2

thanks.

[Djeezus]

Hi,

the number of Vector replicas is only 1 part of the solution. Since Vector fully implements "librdkafka", you can choose whatever "partition.assignment.strategy" you wish in your Vector deployment.

Say you have 10 partitions for "test_topic" : you would then deploy 10 vector instances, with Kafka source config looking something like this :

...
        topics:
          - test_topic
        librdkafka_options:
          partition.assignment.strategy: cooperative-sticky
...

This will ensure that topic is consumed from all 10 partitions in parallel, and even rebalancing is taken into account.
There are obviously other assignment strategies too, but it depends on your kafka setup which to use for best performance.
Also take into account the possible lag due to he Sink, it's hardly ever Kafka that's the bottleneck.

grtz,
Gert

PS : if it's throughput you're looking for, you should not need that high timeouts ;-)

[LinTechSo]

Thanks for the reply.