Skip to content

expose statistics in iceberg-datafusion for fast COUNT() #1428

Open
@debugmiller

Description

@debugmiller

Is your feature request related to a problem or challenge?

Running SELECT COUNT(1) when using iceberg-datafusion results in a table scan. This can be avoided by implementing ExecutionPlan::statistics. Datafusion does this for its built-in parquet scanner by fetching the statistics from parquet metadata when constructing the ExecutionPlan. I was looking to implement this in a similar way (at least for tables without deletes) by iterating over the ManifestEntrys and summing the record_counts. I have a draft PR but wanted to confirm this approach is acceptable before putting in the work to clean it up.

Describe the solution you'd like

count(*) in datafusion does not perform a table scan

Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions