Skip to content

Evaluate IS_NULL at row group and page level in Parquet filtering #20144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open

Evaluate IS_NULL at row group and page level in Parquet filtering #20144

wants to merge 32 commits into from
+349 −223

Conversation

mhaseeb123
Copy link
Member

@mhaseeb123 mhaseeb123 commented Sep 30, 2025 •
edited
Loading

Description

Closes #20125

This PR allows evaluating IS_NULL(col) and by extension NOT(IS_NULL(col)) expressions while filtering Parquet row groups and data pages (requires page index) using corresponding statistics. The PR also includes optimizations in case the host columns containing page-stats don't contain any nulls.

Checklist

  • Optimize page_stats_caster and row_group_stats_caster to use has_is_null_operator
  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@mhaseeb123
Passthrough unary ops through Parquet predicate pushdown
cdc3685
@mhaseeb123
Merge branch 'branch-25.12' into fea/passthrough-unary-ops-in-pq-pred…
da7cf9a 
…icate-pushdown
@mhaseeb123
Update gtests
0d55bd4
@mhaseeb123
Merge branch 'branch-25.12' into fea/passthrough-unary-ops-in-pq-pred…
d7ff34b 
…icate-pushdown
@mhaseeb123
Propagate _always_true to parent expression when expr can't be eval…
204d746 
…uated
@mhaseeb123
Merge branch 'branch-25.12' into fea/passthrough-unary-ops-in-pq-pred…
71c3212 
…icate-pushdown
@mhaseeb123
Merge branch 'branch-25.12' into fea/passthrough-unary-ops-in-pq-pred…
1dfb9a4 
…icate-pushdown
@mhaseeb123
Small update
0638922
@mhaseeb123
Minor updates
0733944
@mhaseeb123
Comments update
c18d5dc
@mhaseeb123
Similar changes in dictionary and bloom filter collectors
88be6f4
@mhaseeb123
Merge branch 'branch-25.12' into fea/passthrough-unary-ops-in-pq-pred…
efa8b16 
…icate-pushdown
@mhaseeb123
Evaluate IS_NULL at rg and page levels in parquet filtering
bd2b32c
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Sep 30, 2025

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions bot added the libcudf Affects libcudf (C++/CUDA) code. label Sep 30, 2025
@mhaseeb123 mhaseeb123 added feature request New feature or request 2 - In Progress Currently a work in progress cuIO cuIO issue non-breaking Non-breaking change libcudf Affects libcudf (C++/CUDA) code. and removed libcudf Affects libcudf (C++/CUDA) code. labels Sep 30, 2025
@mhaseeb123 mhaseeb123 marked this pull request as ready for review October 14, 2025 15:05
@mhaseeb123 mhaseeb123 requested a review from a team as a code owner October 14, 2025 15:05
@mhaseeb123 mhaseeb123 added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress labels Oct 14, 2025
@mhaseeb123 mhaseeb123 mentioned this pull request Oct 14, 2025
Open
2 tasks
PointKernel
PointKernel reviewed Oct 17, 2025
View reviewed changes
Copy link
Member

@PointKernel PointKernel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two quick questions, everything else looks great. Regarding the has_is_null_operator name, it feels a bit wordy, but I couldn’t come up with a better alternative.

@GregoryKimball GregoryKimball moved this to Burndown in libcudf Oct 17, 2025
@mhaseeb123 mhaseeb123 added 4 - Needs Review Waiting for reviewer to review or respond and removed 3 - Ready for Review Ready for review by team labels Oct 21, 2025
mhaseeb123
mhaseeb123 commented Oct 21, 2025
View reviewed changes
mhaseeb123
mhaseeb123 commented Oct 21, 2025
View reviewed changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@PointKernel PointKernel PointKernel approved these changes

@karthikeyann karthikeyann Awaiting requested review from karthikeyann karthikeyann is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

Assignees

@mhaseeb123 mhaseeb123

Labels

4 - Needs Review Waiting for reviewer to review or respond cuIO cuIO issue feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change

Projects

libcudf
Status: Burndown

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[FEA] Support unary operators in Parquet row group and page filtering

2 participants

@mhaseeb123 @PointKernel