Can we expect Parallel Consumer to work with Kafka 4.0.0 client (or 3.9.1)?

[jvissers]

My primary reasons for asking this, is because of Java 24 uptake.

One of the things that will be in that version is the permanent removal of the security manager. We are currently on Java 23, and we have to use the flag -Djava.security.manager=allow to be able to run our service. That option will no longer be available in Java 24, and this requires us to migrate to Kafka 4.0.0, once that hits. Curious whether we are expecting this to be a drop-in replacement, without disruptions to the Parallel Consumer library.

[jvissers]

@astubbs - gentle reminder for the above open question?

[bernardmo]

I'll just add a note for java 24 setup - in some cases it's possible to force kafka-clients lib dependency to 4.0.0 and set the ignoreReflectiveAccessExceptionsForAutoCommitDisabledCheck to true. Not sure what are the side-effects of that kind of setup, but the comment states:

The purpose of the flag is to be a last resort / temporary work-around for changes introduced in newer Kafka Clients that break reflective access and for using wrapped, custom or extended KafkaConsumer classes that fail reflection checks - setting the flag to true will ignore reflection access exceptions during this check.
Note: that library will still try to access auto commit field on the consumer object and if it is accessible and not disabled - the Parallel Consumer will shut down.

[jvissers]

Thanks @bernardmo - at the moment Kafka 3.9.1 is cooking over at Kafka (currently RC1). This version will have a backport of the security manager shim that is needed to run this on Java 24. Do you anticipate that parallel consumer will run unchanged on that specific version?

[bernardmo]

Hi @jvissers.

This version will have a backport of the security manager shim that is needed to run this on Java 24.

That's great to know.

Do you anticipate that parallel consumer will run unchanged on that specific version?

At the moment I have no idea. My comment was placed just to note that the current version of parallel consumer can run on java 24 with kafka clients 4.0.0 (was forced to upgrade the version on one of the projects that I work on due to virtual thread blocking issue). For now I didn't noticed any issues, but again it's a limited use case.