Skip to content

Adjusting batching to only use topic #512

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Mar 25, 2025
Merged

Adjusting batching to only use topic #512

merged 5 commits into from
Mar 25, 2025

Conversation

@Paultagoras
Copy link
Contributor

Summary

  • Added a flag to ignore partition when grouping records (only when exactlyOnce=false)

Checklist

Delete items not relevant to your PR:

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG
  • For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

@Paultagoras Paultagoras requested a review from mzitnik March 24, 2025 15:58
@Paultagoras Paultagoras self-assigned this Mar 24, 2025
mzitnik
mzitnik reviewed Mar 24, 2025
View reviewed changes
Copy link
Collaborator

@mzitnik mzitnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to add some tests for that

@Paultagoras Paultagoras linked an issue Mar 24, 2025 that may be closed by this pull request
Add a feature flag to tweak batching behavior #513
Closed
@Paultagoras Paultagoras marked this pull request as ready for review March 25, 2025 08:10
@mshustov mshustov requested a review from mzitnik March 25, 2025 08:55
mzitnik
mzitnik reviewed Mar 25, 2025
View reviewed changes
src/main/java/com/clickhouse/kafka/connect/sink/ProxySinkTask.java
clickHouseSinkConfig.getDatabase() ))
.collect(Collectors.groupingBy(Record::getTopicAndPartition));
.collect(Collectors.groupingBy(!clickHouseSinkConfig.isExactlyOnce() && clickHouseSinkConfig.isIgnorePartitionsWhenBatching()
? Record::getTopic : Record::getTopicAndPartition));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@mzitnik mzitnik merged commit ea68506 into main Mar 25, 2025
8 checks passed
@mshustov mshustov deleted the adjust-batching branch March 25, 2025 13:49
This was referenced Mar 26, 2025
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mzitnik mzitnik mzitnik approved these changes

Assignees

@Paultagoras Paultagoras

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add a feature flag to tweak batching behavior

2 participants

@Paultagoras @mzitnik