Skip to content

Unify Metadata Handing: use FieldMetadata in Expr::Alias and ExprSchemable #16320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jun 17, 2025
Merged

Unify Metadata Handing: use FieldMetadata in Expr::Alias and ExprSchemable #16320

merged 2 commits into from
Jun 17, 2025

Conversation

alamb
Copy link
Contributor

@alamb alamb commented Jun 7, 2025 •
edited
Loading

Which issue does this PR close?

Rationale for this change

We are working metadata through DataFusion as part of supporting extension types
so having a single unified representation to work on will make the code easier
to do. I added FieldMetadata to avoid the use of BTreeMap direcectly, but found
there were still places that used HashMap to pass metadata around directly

What changes are included in this PR?

Let's use FieldMetadata everywhere

Are these changes tested?

By CI

Are there any user-facing changes?

This is an API change, but I think it sets us up for unified and efficient metadata handling in DataFusion

@github-actions github-actions bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate proto Related to proto crate labels Jun 7, 2025
@alamb alamb mentioned this pull request Jun 7, 2025
Merged
alamb
alamb commented Jun 7, 2025
View reviewed changes
datafusion/expr/src/expr.rs
@@ -3657,7 +3834,7 @@ mod test {
// If this test fails when you change `Expr`, please try
// `Box`ing the fields to make `Expr` smaller
// See https://github.com/apache/datafusion/issues/16199 for details
assert_eq!(size_of::<Expr>(), 144);
assert_eq!(size_of::<Expr>(), 128);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And Expr gets smaller!

@alamb alamb mentioned this pull request Jun 7, 2025
Merged
@alamb
Copy link
Contributor Author

alamb commented Jun 7, 2025

🤖 ./gh_compare_branch_bench.sh Benchmark Script Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/field_metadata2 (b27552a) to 1daa5ed diff
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_field_metadata2
Results will be posted here when complete

@alamb
Copy link
Contributor Author

alamb commented Jun 7, 2025

🤖: Benchmark completed

Details 

group                                         alamb_field_metadata2                  main
-----                                         ---------------------                  ----
logical_aggregate_with_join                   1.00    727.2±5.41µs        ? ?/sec    1.00    726.8±6.09µs        ? ?/sec
logical_select_all_from_1000                  1.00    124.3±0.46ms        ? ?/sec    1.01    125.2±1.46ms        ? ?/sec
logical_select_one_from_700                   1.00    413.5±1.48µs        ? ?/sec    1.00    414.5±1.00µs        ? ?/sec
logical_trivial_join_high_numbered_columns    1.00    376.1±1.74µs        ? ?/sec    1.00    376.7±1.88µs        ? ?/sec
logical_trivial_join_low_numbered_columns     1.00    361.4±2.00µs        ? ?/sec    1.01    364.7±3.44µs        ? ?/sec
physical_intersection                         1.00    833.0±9.70µs        ? ?/sec    1.00    832.3±4.72µs        ? ?/sec
physical_join_consider_sort                   1.00  1382.7±20.89µs        ? ?/sec    1.01  1392.2±14.62µs        ? ?/sec
physical_join_distinct                        1.00    350.2±1.34µs        ? ?/sec    1.01    352.3±1.62µs        ? ?/sec
physical_many_self_joins                      1.00     10.2±0.03ms        ? ?/sec    1.00     10.2±0.02ms        ? ?/sec
physical_plan_clickbench_all                  1.00    142.1±1.85ms        ? ?/sec    1.01    142.9±1.40ms        ? ?/sec
physical_plan_clickbench_q1                   1.00  1694.9±20.79µs        ? ?/sec    1.01  1703.7±23.20µs        ? ?/sec
physical_plan_clickbench_q10                  1.00      2.4±0.02ms        ? ?/sec    1.00      2.4±0.03ms        ? ?/sec
physical_plan_clickbench_q11                  1.01      2.6±0.03ms        ? ?/sec    1.00      2.5±0.01ms        ? ?/sec
physical_plan_clickbench_q12                  1.00      2.7±0.01ms        ? ?/sec    1.01      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q13                  1.02      2.4±0.03ms        ? ?/sec    1.00      2.3±0.01ms        ? ?/sec
physical_plan_clickbench_q14                  1.00      2.5±0.03ms        ? ?/sec    1.00      2.5±0.01ms        ? ?/sec
physical_plan_clickbench_q15                  1.00      2.4±0.04ms        ? ?/sec    1.00      2.4±0.04ms        ? ?/sec
physical_plan_clickbench_q16                  1.00      2.3±0.03ms        ? ?/sec    1.00      2.3±0.03ms        ? ?/sec
physical_plan_clickbench_q17                  1.01      2.4±0.03ms        ? ?/sec    1.00      2.4±0.02ms        ? ?/sec
physical_plan_clickbench_q18                  1.01  1961.5±17.48µs        ? ?/sec    1.00  1948.4±23.03µs        ? ?/sec
physical_plan_clickbench_q19                  1.00      2.9±0.03ms        ? ?/sec    1.01      2.9±0.04ms        ? ?/sec
physical_plan_clickbench_q2                   1.00  1914.2±19.20µs        ? ?/sec    1.00  1918.1±28.03µs        ? ?/sec
physical_plan_clickbench_q20                  1.01  1670.0±14.37µs        ? ?/sec    1.00  1652.7±31.38µs        ? ?/sec
physical_plan_clickbench_q21                  1.02  1998.1±32.35µs        ? ?/sec    1.00  1949.9±19.90µs        ? ?/sec
physical_plan_clickbench_q22                  1.00      2.6±0.02ms        ? ?/sec    1.00      2.5±0.02ms        ? ?/sec
physical_plan_clickbench_q23                  1.00      2.8±0.03ms        ? ?/sec    1.01      2.8±0.02ms        ? ?/sec
physical_plan_clickbench_q24                  1.00      4.5±0.03ms        ? ?/sec    1.05      4.7±0.04ms        ? ?/sec
physical_plan_clickbench_q25                  1.00      2.0±0.03ms        ? ?/sec    1.00      2.0±0.02ms        ? ?/sec
physical_plan_clickbench_q26                  1.00  1839.7±19.85µs        ? ?/sec    1.00  1833.7±28.51µs        ? ?/sec
physical_plan_clickbench_q27                  1.00      2.0±0.02ms        ? ?/sec    1.00      2.1±0.02ms        ? ?/sec
physical_plan_clickbench_q28                  1.00      2.8±0.05ms        ? ?/sec    1.00      2.8±0.03ms        ? ?/sec
physical_plan_clickbench_q29                  1.00      3.5±0.06ms        ? ?/sec    1.00      3.5±0.04ms        ? ?/sec
physical_plan_clickbench_q3                   1.00  1900.5±24.04µs        ? ?/sec    1.00  1894.8±15.69µs        ? ?/sec
physical_plan_clickbench_q30                  1.00     12.9±0.10ms        ? ?/sec    1.01     13.1±0.07ms        ? ?/sec
physical_plan_clickbench_q31                  1.00      2.8±0.05ms        ? ?/sec    1.01      2.9±0.02ms        ? ?/sec
physical_plan_clickbench_q32                  1.00      2.8±0.04ms        ? ?/sec    1.02      2.9±0.06ms        ? ?/sec
physical_plan_clickbench_q33                  1.00      2.4±0.02ms        ? ?/sec    1.00      2.4±0.02ms        ? ?/sec
physical_plan_clickbench_q34                  1.01      2.1±0.03ms        ? ?/sec    1.00      2.1±0.03ms        ? ?/sec
physical_plan_clickbench_q35                  1.01      2.2±0.03ms        ? ?/sec    1.00      2.2±0.03ms        ? ?/sec
physical_plan_clickbench_q36                  1.00      2.9±0.03ms        ? ?/sec    1.01      2.9±0.02ms        ? ?/sec
physical_plan_clickbench_q37                  1.00      2.9±0.04ms        ? ?/sec    1.01      2.9±0.02ms        ? ?/sec
physical_plan_clickbench_q38                  1.00      2.9±0.05ms        ? ?/sec    1.02      2.9±0.03ms        ? ?/sec
physical_plan_clickbench_q39                  1.00      2.7±0.02ms        ? ?/sec    1.02      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q4                   1.02  1658.4±23.12µs        ? ?/sec    1.00  1631.8±20.27µs        ? ?/sec
physical_plan_clickbench_q40                  1.00      3.3±0.03ms        ? ?/sec    1.01      3.4±0.04ms        ? ?/sec
physical_plan_clickbench_q41                  1.00      2.9±0.04ms        ? ?/sec    1.01      2.9±0.03ms        ? ?/sec
physical_plan_clickbench_q42                  1.00      2.8±0.04ms        ? ?/sec    1.02      2.9±0.07ms        ? ?/sec
physical_plan_clickbench_q43                  1.00      3.2±0.03ms        ? ?/sec    1.01      3.2±0.02ms        ? ?/sec
physical_plan_clickbench_q44                  1.00  1792.2±25.27µs        ? ?/sec    1.00  1794.5±23.47µs        ? ?/sec
physical_plan_clickbench_q45                  1.00  1797.3±19.71µs        ? ?/sec    1.02  1824.3±22.44µs        ? ?/sec
physical_plan_clickbench_q46                  1.00      2.2±0.02ms        ? ?/sec    1.00      2.2±0.04ms        ? ?/sec
physical_plan_clickbench_q47                  1.00      2.8±0.04ms        ? ?/sec    1.00      2.8±0.03ms        ? ?/sec
physical_plan_clickbench_q48                  1.00      3.4±0.03ms        ? ?/sec    1.00      3.4±0.02ms        ? ?/sec
physical_plan_clickbench_q49                  1.00      3.6±0.03ms        ? ?/sec    1.00      3.6±0.02ms        ? ?/sec
physical_plan_clickbench_q5                   1.01  1854.9±17.31µs        ? ?/sec    1.00  1837.1±21.00µs        ? ?/sec
physical_plan_clickbench_q50                  1.00      3.2±0.02ms        ? ?/sec    1.02      3.3±0.03ms        ? ?/sec
physical_plan_clickbench_q51                  1.00      2.3±0.01ms        ? ?/sec    1.03      2.3±0.04ms        ? ?/sec
physical_plan_clickbench_q52                  1.00      3.1±0.03ms        ? ?/sec    1.00      3.1±0.03ms        ? ?/sec
physical_plan_clickbench_q6                   1.01  1870.5±26.82µs        ? ?/sec    1.00  1844.3±24.22µs        ? ?/sec
physical_plan_clickbench_q7                   1.02  1721.4±21.85µs        ? ?/sec    1.00  1679.6±25.98µs        ? ?/sec
physical_plan_clickbench_q8                   1.00      2.4±0.02ms        ? ?/sec    1.00      2.4±0.03ms        ? ?/sec
physical_plan_clickbench_q9                   1.00      2.3±0.02ms        ? ?/sec    1.01      2.3±0.02ms        ? ?/sec
physical_plan_tpcds_all                       1.00   1044.2±5.58ms        ? ?/sec    1.01   1058.2±4.78ms        ? ?/sec
physical_plan_tpch_all                        1.00     62.7±0.42ms        ? ?/sec    1.02     63.7±0.20ms        ? ?/sec
physical_plan_tpch_q1                         1.00      2.1±0.03ms        ? ?/sec    1.03      2.2±0.10ms        ? ?/sec
physical_plan_tpch_q10                        1.00      3.8±0.01ms        ? ?/sec    1.01      3.9±0.03ms        ? ?/sec
physical_plan_tpch_q11                        1.00      3.3±0.01ms        ? ?/sec    1.01      3.3±0.01ms        ? ?/sec
physical_plan_tpch_q12                        1.00  1831.7±28.79µs        ? ?/sec    1.02  1875.7±21.88µs        ? ?/sec
physical_plan_tpch_q13                        1.00  1459.1±10.18µs        ? ?/sec    1.02  1490.2±18.00µs        ? ?/sec
physical_plan_tpch_q14                        1.00  1965.8±10.69µs        ? ?/sec    1.01   1985.0±8.07µs        ? ?/sec
physical_plan_tpch_q16                        1.00      2.5±0.01ms        ? ?/sec    1.03      2.5±0.00ms        ? ?/sec
physical_plan_tpch_q17                        1.00      2.4±0.01ms        ? ?/sec    1.03      2.5±0.05ms        ? ?/sec
physical_plan_tpch_q18                        1.00      2.7±0.01ms        ? ?/sec    1.02      2.7±0.00ms        ? ?/sec
physical_plan_tpch_q19                        1.00      3.4±0.02ms        ? ?/sec    1.04      3.5±0.02ms        ? ?/sec
physical_plan_tpch_q2                         1.00      5.5±0.01ms        ? ?/sec    1.02      5.6±0.06ms        ? ?/sec
physical_plan_tpch_q20                        1.00      3.1±0.02ms        ? ?/sec    1.03      3.2±0.05ms        ? ?/sec
physical_plan_tpch_q21                        1.00      4.2±0.07ms        ? ?/sec    1.01      4.2±0.05ms        ? ?/sec
physical_plan_tpch_q22                        1.00      2.8±0.05ms        ? ?/sec    1.02      2.8±0.03ms        ? ?/sec
physical_plan_tpch_q3                         1.00      2.6±0.02ms        ? ?/sec    1.01      2.6±0.01ms        ? ?/sec
physical_plan_tpch_q4                         1.00  1594.3±16.78µs        ? ?/sec    1.02  1618.3±22.77µs        ? ?/sec
physical_plan_tpch_q5                         1.00      3.2±0.04ms        ? ?/sec    1.01      3.2±0.01ms        ? ?/sec
physical_plan_tpch_q6                         1.00    869.5±4.44µs        ? ?/sec    1.02    889.6±5.24µs        ? ?/sec
physical_plan_tpch_q7                         1.00      4.2±0.02ms        ? ?/sec    1.02      4.3±0.06ms        ? ?/sec
physical_plan_tpch_q8                         1.00      5.2±0.02ms        ? ?/sec    1.01      5.3±0.01ms        ? ?/sec
physical_plan_tpch_q9                         1.00      4.1±0.04ms        ? ?/sec    1.01      4.1±0.02ms        ? ?/sec
physical_select_aggregates_from_200           1.00     20.8±0.09ms        ? ?/sec    1.01     21.0±0.06ms        ? ?/sec
physical_select_all_from_1000                 1.00    137.9±0.40ms        ? ?/sec    1.01    139.6±1.15ms        ? ?/sec
physical_select_one_from_700                  1.00   1060.6±5.17µs        ? ?/sec    1.00   1059.5±3.09µs        ? ?/sec
physical_sorted_union_orderby                 1.00     42.8±0.49ms        ? ?/sec    1.02     43.8±0.39ms        ? ?/sec
physical_theta_join_consider_sort             1.00   1742.1±8.06µs        ? ?/sec    1.01  1758.5±11.59µs        ? ?/sec
physical_unnest_to_join                       1.00  1303.8±10.65µs        ? ?/sec    1.01   1317.9±9.27µs        ? ?/sec
with_param_values_many_columns                1.00    129.2±1.21µs        ? ?/sec    1.10    141.9±1.05µs        ? ?/sec

@github-actions github-actions bot added the documentation Improvements or additions to documentation label Jun 8, 2025
@alamb alamb force-pushed the alamb/field_metadata2 branch from ec3c2dd to 3a7c9c8 Compare June 11, 2025 04:10
@alamb alamb marked this pull request as ready for review June 11, 2025 04:10
@github-actions github-actions bot removed the optimizer Optimizer rules label Jun 11, 2025
alamb
alamb commented Jun 11, 2025
View reviewed changes
datafusion/expr/src/expr.rs
@@ -601,7 +625,7 @@ pub struct Alias {
pub expr: Box<Expr>,
pub relation: Option<TableReference>,
pub name: String,
pub metadata: Option<std::collections::HashMap<String, String>>,
pub metadata: Option<FieldMetadata>,
Copy link
Contributor Author

@alamb alamb Jun 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is the point of the PR is to use the FieldMetadata struct introduced in this PR everywhere:

The rest of the changes in this PR are consequences of this change

@alamb
Copy link
Contributor Author

alamb commented Jun 11, 2025

🤖 ./gh_compare_branch_bench.sh Benchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/field_metadata2 (3a7c9c8) to 1d61f31 diff
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_field_metadata2
Results will be posted here when complete

@alamb
Copy link
Contributor Author

alamb commented Jun 11, 2025

🤖: Benchmark completed

Details 

group                                         alamb_field_metadata2                  main
-----                                         ---------------------                  ----
logical_aggregate_with_join                   1.02    729.6±3.11µs        ? ?/sec    1.00    716.1±3.21µs        ? ?/sec
logical_select_all_from_1000                  1.00    123.8±0.46ms        ? ?/sec    1.01    125.1±0.26ms        ? ?/sec
logical_select_one_from_700                   1.03    419.2±2.70µs        ? ?/sec    1.00    408.8±2.44µs        ? ?/sec
logical_trivial_join_high_numbered_columns    1.02    377.3±2.28µs        ? ?/sec    1.00    369.5±1.77µs        ? ?/sec
logical_trivial_join_low_numbered_columns     1.03    363.4±1.60µs        ? ?/sec    1.00    354.1±1.70µs        ? ?/sec
physical_intersection                         1.02    835.9±5.05µs        ? ?/sec    1.00    822.0±6.34µs        ? ?/sec
physical_join_consider_sort                   1.02   1379.8±6.90µs        ? ?/sec    1.00   1354.7±5.39µs        ? ?/sec
physical_join_distinct                        1.03    353.3±1.61µs        ? ?/sec    1.00    344.5±1.38µs        ? ?/sec
physical_many_self_joins                      1.03     10.3±0.04ms        ? ?/sec    1.00     10.0±0.04ms        ? ?/sec
physical_plan_clickbench_all                  1.00    139.7±1.30ms        ? ?/sec    1.03    144.0±1.47ms        ? ?/sec
physical_plan_clickbench_q1                   1.00  1697.7±18.59µs        ? ?/sec    1.00  1694.5±26.06µs        ? ?/sec
physical_plan_clickbench_q10                  1.00      2.4±0.02ms        ? ?/sec    1.01      2.5±0.03ms        ? ?/sec
physical_plan_clickbench_q11                  1.00      2.5±0.02ms        ? ?/sec    1.03      2.6±0.03ms        ? ?/sec
physical_plan_clickbench_q12                  1.00      2.6±0.02ms        ? ?/sec    1.02      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q13                  1.00      2.3±0.03ms        ? ?/sec    1.01      2.4±0.02ms        ? ?/sec
physical_plan_clickbench_q14                  1.00      2.5±0.02ms        ? ?/sec    1.02      2.6±0.03ms        ? ?/sec
physical_plan_clickbench_q15                  1.00      2.4±0.02ms        ? ?/sec    1.01      2.4±0.02ms        ? ?/sec
physical_plan_clickbench_q16                  1.00      2.3±0.02ms        ? ?/sec    1.02      2.3±0.02ms        ? ?/sec
physical_plan_clickbench_q17                  1.00      2.4±0.01ms        ? ?/sec    1.03      2.4±0.03ms        ? ?/sec
physical_plan_clickbench_q18                  1.00  1933.0±18.29µs        ? ?/sec    1.02  1966.0±19.30µs        ? ?/sec
physical_plan_clickbench_q19                  1.00      2.8±0.03ms        ? ?/sec    1.03      2.9±0.03ms        ? ?/sec
physical_plan_clickbench_q2                   1.00  1904.1±23.73µs        ? ?/sec    1.01  1917.6±22.98µs        ? ?/sec
physical_plan_clickbench_q20                  1.00  1622.7±14.08µs        ? ?/sec    1.03  1663.9±18.50µs        ? ?/sec
physical_plan_clickbench_q21                  1.00  1932.7±27.82µs        ? ?/sec    1.03      2.0±0.02ms        ? ?/sec
physical_plan_clickbench_q22                  1.00      2.5±0.02ms        ? ?/sec    1.02      2.6±0.02ms        ? ?/sec
physical_plan_clickbench_q23                  1.00      2.8±0.02ms        ? ?/sec    1.03      2.9±0.03ms        ? ?/sec
physical_plan_clickbench_q24                  1.00      4.5±0.02ms        ? ?/sec    1.05      4.7±0.04ms        ? ?/sec
physical_plan_clickbench_q25                  1.00  1989.6±21.12µs        ? ?/sec    1.02      2.0±0.02ms        ? ?/sec
physical_plan_clickbench_q26                  1.00  1779.7±13.43µs        ? ?/sec    1.04  1849.8±20.58µs        ? ?/sec
physical_plan_clickbench_q27                  1.00      2.0±0.02ms        ? ?/sec    1.03      2.1±0.03ms        ? ?/sec
physical_plan_clickbench_q28                  1.00      2.7±0.02ms        ? ?/sec    1.03      2.8±0.03ms        ? ?/sec
physical_plan_clickbench_q29                  1.00      3.4±0.03ms        ? ?/sec    1.04      3.6±0.05ms        ? ?/sec
physical_plan_clickbench_q3                   1.01  1898.1±14.34µs        ? ?/sec    1.00  1887.6±16.67µs        ? ?/sec
physical_plan_clickbench_q30                  1.00     12.7±0.09ms        ? ?/sec    1.04     13.2±0.11ms        ? ?/sec
physical_plan_clickbench_q31                  1.00      2.8±0.03ms        ? ?/sec    1.03      2.9±0.02ms        ? ?/sec
physical_plan_clickbench_q32                  1.00      2.8±0.03ms        ? ?/sec    1.03      2.9±0.03ms        ? ?/sec
physical_plan_clickbench_q33                  1.00      2.4±0.02ms        ? ?/sec    1.03      2.5±0.03ms        ? ?/sec
physical_plan_clickbench_q34                  1.00      2.1±0.03ms        ? ?/sec    1.02      2.1±0.02ms        ? ?/sec
physical_plan_clickbench_q35                  1.00      2.2±0.07ms        ? ?/sec    1.00      2.2±0.02ms        ? ?/sec
physical_plan_clickbench_q36                  1.00      2.9±0.03ms        ? ?/sec    1.04      3.0±0.06ms        ? ?/sec
physical_plan_clickbench_q37                  1.00      2.9±0.05ms        ? ?/sec    1.03      3.0±0.05ms        ? ?/sec
physical_plan_clickbench_q38                  1.00      2.9±0.02ms        ? ?/sec    1.03      3.0±0.04ms        ? ?/sec
physical_plan_clickbench_q39                  1.00      2.7±0.02ms        ? ?/sec    1.04      2.8±0.04ms        ? ?/sec
physical_plan_clickbench_q4                   1.00  1650.6±22.94µs        ? ?/sec    1.00  1652.9±16.05µs        ? ?/sec
physical_plan_clickbench_q40                  1.00      3.3±0.03ms        ? ?/sec    1.04      3.4±0.06ms        ? ?/sec
physical_plan_clickbench_q41                  1.00      2.8±0.02ms        ? ?/sec    1.05      3.0±0.04ms        ? ?/sec
physical_plan_clickbench_q42                  1.00      2.8±0.02ms        ? ?/sec    1.04      2.9±0.03ms        ? ?/sec
physical_plan_clickbench_q43                  1.00      3.1±0.03ms        ? ?/sec    1.04      3.2±0.04ms        ? ?/sec
physical_plan_clickbench_q44                  1.00  1771.7±18.76µs        ? ?/sec    1.03  1824.2±31.49µs        ? ?/sec
physical_plan_clickbench_q45                  1.00  1783.9±14.02µs        ? ?/sec    1.02  1824.0±22.37µs        ? ?/sec
physical_plan_clickbench_q46                  1.00      2.2±0.03ms        ? ?/sec    1.02      2.2±0.02ms        ? ?/sec
physical_plan_clickbench_q47                  1.00      2.7±0.03ms        ? ?/sec    1.03      2.8±0.03ms        ? ?/sec
physical_plan_clickbench_q48                  1.00      3.3±0.03ms        ? ?/sec    1.04      3.5±0.02ms        ? ?/sec
physical_plan_clickbench_q49                  1.00      3.6±0.04ms        ? ?/sec    1.04      3.7±0.04ms        ? ?/sec
physical_plan_clickbench_q5                   1.01  1844.2±15.95µs        ? ?/sec    1.00  1829.6±18.24µs        ? ?/sec
physical_plan_clickbench_q50                  1.00      3.1±0.02ms        ? ?/sec    1.07      3.4±0.07ms        ? ?/sec
physical_plan_clickbench_q51                  1.00      2.2±0.02ms        ? ?/sec    1.04      2.3±0.02ms        ? ?/sec
physical_plan_clickbench_q52                  1.00      3.0±0.02ms        ? ?/sec    1.04      3.2±0.03ms        ? ?/sec
physical_plan_clickbench_q6                   1.00  1856.5±16.15µs        ? ?/sec    1.00  1852.9±22.15µs        ? ?/sec
physical_plan_clickbench_q7                   1.02  1701.7±20.36µs        ? ?/sec    1.00  1674.9±13.35µs        ? ?/sec
physical_plan_clickbench_q8                   1.00      2.4±0.03ms        ? ?/sec    1.00      2.4±0.03ms        ? ?/sec
physical_plan_clickbench_q9                   1.00      2.3±0.02ms        ? ?/sec    1.02      2.3±0.02ms        ? ?/sec
physical_plan_tpcds_all                       1.00   1057.3±5.10ms        ? ?/sec    1.02   1073.7±4.94ms        ? ?/sec
physical_plan_tpch_all                        1.00     62.7±0.23ms        ? ?/sec    1.03     64.4±0.42ms        ? ?/sec
physical_plan_tpch_q1                         1.00      2.1±0.01ms        ? ?/sec    1.03      2.2±0.01ms        ? ?/sec
physical_plan_tpch_q10                        1.00      3.8±0.02ms        ? ?/sec    1.02      3.9±0.01ms        ? ?/sec
physical_plan_tpch_q11                        1.00      3.3±0.02ms        ? ?/sec    1.02      3.3±0.02ms        ? ?/sec
physical_plan_tpch_q12                        1.00  1817.9±10.09µs        ? ?/sec    1.03  1875.5±11.45µs        ? ?/sec
physical_plan_tpch_q13                        1.00   1455.3±7.30µs        ? ?/sec    1.02   1479.2±6.88µs        ? ?/sec
physical_plan_tpch_q14                        1.00  1946.8±10.05µs        ? ?/sec    1.02   1982.0±9.71µs        ? ?/sec
physical_plan_tpch_q16                        1.00      2.5±0.01ms        ? ?/sec    1.03      2.5±0.01ms        ? ?/sec
physical_plan_tpch_q17                        1.00      2.4±0.02ms        ? ?/sec    1.02      2.4±0.02ms        ? ?/sec
physical_plan_tpch_q18                        1.00      2.7±0.02ms        ? ?/sec    1.02      2.7±0.01ms        ? ?/sec
physical_plan_tpch_q19                        1.00      3.4±0.02ms        ? ?/sec    1.05      3.5±0.02ms        ? ?/sec
physical_plan_tpch_q2                         1.00      5.4±0.03ms        ? ?/sec    1.02      5.5±0.01ms        ? ?/sec
physical_plan_tpch_q20                        1.00      3.1±0.03ms        ? ?/sec    1.03      3.2±0.02ms        ? ?/sec
physical_plan_tpch_q21                        1.00      4.1±0.01ms        ? ?/sec    1.03      4.2±0.02ms        ? ?/sec
physical_plan_tpch_q22                        1.00      2.7±0.01ms        ? ?/sec    1.04      2.8±0.03ms        ? ?/sec
physical_plan_tpch_q3                         1.00      2.5±0.01ms        ? ?/sec    1.02      2.6±0.01ms        ? ?/sec
physical_plan_tpch_q4                         1.00  1580.0±11.14µs        ? ?/sec    1.03  1621.1±13.42µs        ? ?/sec
physical_plan_tpch_q5                         1.00      3.2±0.02ms        ? ?/sec    1.01      3.2±0.01ms        ? ?/sec
physical_plan_tpch_q6                         1.00    865.5±5.79µs        ? ?/sec    1.03    892.0±5.96µs        ? ?/sec
physical_plan_tpch_q7                         1.00      4.2±0.01ms        ? ?/sec    1.02      4.3±0.02ms        ? ?/sec
physical_plan_tpch_q8                         1.00      5.2±0.03ms        ? ?/sec    1.02      5.2±0.04ms        ? ?/sec
physical_plan_tpch_q9                         1.00      4.1±0.01ms        ? ?/sec    1.01      4.1±0.01ms        ? ?/sec
physical_select_aggregates_from_200           1.00     20.9±0.07ms        ? ?/sec    1.00     20.9±0.07ms        ? ?/sec
physical_select_all_from_1000                 1.00    138.3±0.31ms        ? ?/sec    1.01    139.3±0.29ms        ? ?/sec
physical_select_one_from_700                  1.03  1072.7±16.08µs        ? ?/sec    1.00   1036.5±4.58µs        ? ?/sec
physical_sorted_union_orderby                 1.00     43.3±0.25ms        ? ?/sec    1.01     43.8±0.30ms        ? ?/sec
physical_theta_join_consider_sort             1.01   1746.9±7.45µs        ? ?/sec    1.00   1722.5±6.42µs        ? ?/sec
physical_unnest_to_join                       1.02  1304.1±25.48µs        ? ?/sec    1.00   1283.8±7.45µs        ? ?/sec
with_param_values_many_columns                1.00    127.9±1.43µs        ? ?/sec    1.10    140.6±0.95µs        ? ?/sec

paleolimbot
paleolimbot approved these changes Jun 12, 2025
View reviewed changes
Copy link
Member

@paleolimbot paleolimbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! The metadata we're passing around can be up to a few kilobytes and this (and the previous PR) are great quality of life improvements.

timsaucer
timsaucer approved these changes Jun 12, 2025
View reviewed changes
Copy link
Contributor

@timsaucer timsaucer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice addition. Thank you.

@alamb alamb merged commit 0c3bb78 into apache:main Jun 17, 2025
52 of 53 checks passed
@alamb
Copy link
Contributor Author

alamb commented Jun 17, 2025

Thanks @timsaucer and @paleolimbot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@paleolimbot paleolimbot paleolimbot approved these changes

@timsaucer timsaucer timsaucer approved these changes

Assignees
No one assigned
Labels
core Core DataFusion crate documentation Improvements or additions to documentation logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates proto Related to proto crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@alamb @paleolimbot @timsaucer