Modifying IPFIXcol2 Kafka Output Plugin to Support Multiple Topics

[Lidhoria]

I hope you are doing well.

I have been working with the IPFIXcol2 collector and its Kafka output plugin as defined in the startup.xml configuration file. Currently, I can specify a single Kafka topic in the configuration, but I am interested in extending this functionality to support two separate topics within Kafka.

Could you please clarify the following:

Is it currently possible to define multiple Kafka topics in the configuration file under the Kafka output plugin?
If not, would it be feasible to modify the IPFIXcol2 code to support two separate topics within Kafka? If so, could you provide guidance on the best approach to implement this?
Are there any potential limitations or concerns regarding such a modification?
Your insights and suggestions would be greatly appreciated. Thank you for your time and support!

[Lukas955]

Hi,

I think it should be enough to just list another output that will have a different topic. For example:

    ...

    <outputs>
        <!-- First instance -->
        <kafka>
            <name>Send to Kafka T1</name>
            <brokers>127.0.0.1</brokers>
            <topic>ipfix_one</topic>
            <blocking>false</blocking>
            <partition>unassigned</partition>

            <!-- Zero or more additional properties -->
            <property>
                <key>compression.codec</key>
                <value>lz4</value>
            </property>
        </kafka>

        <!-- Second instance -->
        <kafka>
            <name>Send to Kafka T2</name>
            <brokers>127.0.0.1</brokers>
            <topic>ipfix_two</topic>
            <blocking>false</blocking>
            <partition>unassigned</partition>

            <!-- Zero or more additional properties -->
            <property>
                <key>compression.codec</key>
                <value>lz4</value>
            </property>
        </kafka>
    </outputs>

    ...

Note: It seems that the librdkafka API itself, which is used to communicate with Apache Kafka, does not allow to send one message to multiple topics at the same time. The above configuration will create two separate connections, with each producing under its own topic.

Lukas