-
Notifications
You must be signed in to change notification settings - Fork 980
Open
Labels
feature requestNew feature or requestNew feature or request
Milestone
Description
Is your feature request related to a problem? Please describe.
Currently, we only support binary operators such as >, <, == to filter Parquet row groups and data pages. We would now like to extend our parquet filtering to unary operators such as IS_NULL. Currently, using a unary operator in the AST expression leads to a runtime error in the predicate pushdown code (first occurrence: stats based row group filtering).
Describe the solution you'd like
- Pass through unary operators through current Parquet predicate pushdown filters without throwing an error (PR Passthrough unary ops through Parquet predicate pushdown #20127)
- Support common unary operators such as
IS_NULLin Parquet predicate pushdown (PR EvaluateIS_NULLat row group and page level in Parquet filtering #20144)
Describe alternatives you've considered
Pre-processing the AST expression to not use unary operators during read_parquet and apply the remaining filters post-read
Additional context
Helps Velox
Metadata
Metadata
Assignees
Labels
feature requestNew feature or requestNew feature or request
Type
Projects
Status
In progress