Producer support for per-topic configs #449
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nijikon
reviewed
May 7, 2024
7 tasks
mensfeld
added a commit
to karafka/karafka-rdkafka
that referenced
this pull request
May 8, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces the ability to use topic-specific custom settings as defined here: https://karafka.io/docs/Librdkafka-Configuration/ on a per-message dispatch basis.
This allows one producer to have different
ackslevels specific to topics and other custom settings.In addition, it caches the rdkafka topic object so we do not have to create it with each request, saving on memory allocations. Since
with_inneris locking, we do not have to worry about race conditions when adding configs to the cache.We use the config
hashvalue, so the same config created from different Ruby hash objects will reuse the same cache entry.We allow multiple configs per the same topic because I've stumbled upon a case where the user had different importance messages delivered to the same topic and wanted to tune this. Makes sense to me.
close #436