-
Notifications
You must be signed in to change notification settings - Fork 535
[VL] Check whether the write is a supported iceberg write before executing offload #10900
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Run Gluten ClickHouse CI on ARM |
I found that with the current offload + validate fallback: |
|
||
object OffloadIcebergWrite { | ||
|
||
def supportsWrite(write: Write): Boolean = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added reflection here to ensure that this rule can still execute correctly when there's no Iceberg dependency. A similar approach is used in offloadBatchScan.
Yes, this sounds like one of the major issues we have for making different components (e.g., paimon + iceberg) work together. This validator was written for keeping up with some legacy code in our query planner, and I do think it should be eventually removed or redesigned. We should make one component try to offload a plan node that the previous component couldn't handle, not skip it. Maybe we can just make an attempt to remove that validator to see what happens then. |
Run Gluten ClickHouse CI on ARM |
I see, this sounds great!
Do we have a plan for this part. |
I haven't had a clear plan on it, perhaps I can take a closer look next week or so. Please feel free to take it on if you are interested. |
Run Gluten ClickHouse CI on ARM |
What changes are proposed in this pull request?
Check whether the write is a supported iceberg write before executing
How was this patch tested?