Questions on integrating with existing KafkaProducer-based app and usage without object storage

[ankit-khare-2015]

HI Alexander Alten-Lorenz

First of all, thank you for building this connector it looks quite simple with high value.

I had a few questions while evaluating the integration for our setup:

1. KafkaProducer Compatibility
   Currently, our application uses the native KafkaProducer (from org.apache.kafka.clients.producer) and not Confluent's implementation. In this case, would I need to change our existing producer implementation to use this connector (since it's based on the Confluent stack)? Or can it work independently without switching the producer library?

2. Use Case without Object Storage
   Currently, we do not use any object storage, such as S3. Instead, we write all Kafka messages directly into PostgreSQL. However, we're now exploring ways to handle large messages (greater than 1 MB), which Kafka tends to struggle with.

   • Would you recommend using this connector as a complete switch for handling such payloads (e.g., storing them externally and passing references)?
   • Or should I implement custom exception handling to catch oversized payloads and offload only those?

3. Docker Integration
   I'm currently working on containerizing this setup.

Thanks again for your efforts on this tool. Looking forward to your insights!

Best regards,
Ankit

[2pk03]

Hi @ankit-khare-2015,

For 1 - it should run natively. I used the Confluent since the class deals better with Kerberos and SSL. But tbh, I could not test with a native kafka, would be awesome when you could do that!

I'd recommend adding a handler which detects either large messages, or you reconfigure the topics and stream larger messages into a new topic, like "oversized" and use the connector for all messages in that topic and store them into S3. If you have HDFS, you could add an HDFS handler, like when you're in a CDH stack.

Yeah a docker would be great! Pls PR and I'll approve ASAP!

thanks, best,
--alex

[ankit-khare-2015]

Hi Alex,

I have been working on a few projects that I wanted to discuss.

I will start with this project soon:

1. First, I will add Continuous Integration (CI) to ensure I don't break any existing code and to make sure tests run smoothly.
2. I will include an example using Docker and MinIO to help us better understand how this will work in real time.
3. We can also discuss different handlers further.

Looking forward to your thoughts!