Skip to content

feat: mapping sql Char/Text/String default to Utf8View #16290

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Jun 17, 2025
Merged

feat: mapping sql Char/Text/String default to Utf8View #16290

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
14bd404
feat: mapping sql Char/Text/String default to Utf8View
zhuqi-lucas Jun 6, 2025
4973bf9
Add support utf8view for sort merge join
zhuqi-lucas Jun 6, 2025
85bf391
fix binary utf8view union with int32
zhuqi-lucas Jun 8, 2025
406575b
fix slt order
zhuqi-lucas Jun 8, 2025
bcb5645
Merge remote-tracking branch 'upstream/main' into issue_16288
zhuqi-lucas Jun 8, 2025
8ad6491
fix
zhuqi-lucas Jun 8, 2025
0b4ee41
fmt
zhuqi-lucas Jun 8, 2025
8ee72c9
Fix test
zhuqi-lucas Jun 8, 2025
0b05cb6
fix
zhuqi-lucas Jun 11, 2025
e4560a1
Merge remote-tracking branch 'upstream/main' into issue_16288
zhuqi-lucas Jun 11, 2025
d825fd1
fix test
zhuqi-lucas Jun 11, 2025
f341e1e
fix
zhuqi-lucas Jun 11, 2025
6485228
clean
zhuqi-lucas Jun 11, 2025
fd20268
Address comments
zhuqi-lucas Jun 11, 2025
a715d10
Merge remote-tracking branch 'upstream/main' into issue_16288
zhuqi-lucas Jun 11, 2025
15aba27
Merge remote-tracking branch 'upstream/main' into issue_16288
zhuqi-lucas Jun 16, 2025
9802850
Fix test
zhuqi-lucas Jun 16, 2025
1187834
support md5 for ut8view
zhuqi-lucas Jun 17, 2025
69838fd
Merge remote-tracking branch 'upstream/main' into issue_16288
zhuqi-lucas Jun 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions datafusion/common/src/config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,11 @@ config_namespace! {
/// Default is false.
pub map_varchar_to_utf8view: bool, default = true

/// If true, `CHAR` and `Text` and `String` is mapped to `Utf8View` during SQL planning.
/// If false, `CHAR` and `Text` and `String` is mapped to `Utf8` during SQL planning.
/// Default is true.
pub map_char_to_utf8view: bool, default = true

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need the two configs? I mean, if we can directly use one.

Imagine that one is true, and another is false, does it make sense? Before we didn't have utf8view, varchar,char, text, string all will be utf8, they're consistent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @xudong963 for review, this is true, we can use one config, maybe the name should be:

map_sql_string_to_utf8view for unifed all sql string to utf8view

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about:

/// If true, string types (VARCHAR, CHAR, Text, and String) are mapped to `Utf8View` during SQL planning.
/// If false, they are mapped to `Utf8`.
/// Default is true.
pub map_string_types_to_utf8view: bool, default = true

Also cc @alamb

Copy link
Contributor Author

@zhuqi-lucas zhuqi-lucas Jun 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about:

/// If true, string types (VARCHAR, CHAR, Text, and String) are mapped to `Utf8View` during SQL planning.
/// If false, they are mapped to `Utf8`.
/// Default is true.
pub map_string_types_to_utf8view: bool, default = true

Also cc @alamb

Thanks @xudong963 looks good to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in latest PR, thanks!

/// When set to true, the source locations relative to the original SQL
/// query (i.e. [`Span`](https://docs.rs/sqlparser/latest/sqlparser/tokenizer/struct.Span.html)) will be collected
/// and recorded in the logical plan nodes.
Expand Down
1 change: 1 addition & 0 deletions datafusion/core/src/execution/session_state.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -495,6 +495,7 @@ impl SessionState {
.enable_options_value_normalization,
support_varchar_with_length: sql_parser_options.support_varchar_with_length,
map_varchar_to_utf8view: sql_parser_options.map_varchar_to_utf8view,
map_char_to_utf8view: sql_parser_options.map_char_to_utf8view,
collect_spans: sql_parser_options.collect_spans,
}
}
Expand Down
2 changes: 1 addition & 1 deletion datafusion/core/tests/sql/create_drop.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ async fn create_external_table_with_ddl() -> Result<()> {
assert_eq!(3, table_schema.fields().len());

assert_eq!(&DataType::Int32, table_schema.field(0).data_type());
assert_eq!(&DataType::Utf8, table_schema.field(1).data_type());
assert_eq!(&DataType::Utf8View, table_schema.field(1).data_type());
assert_eq!(&DataType::Boolean, table_schema.field(2).data_type());

Ok(())
Expand Down
2 changes: 1 addition & 1 deletion datafusion/core/tests/user_defined/user_defined_scalar_functions.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1181,7 +1181,7 @@ async fn create_scalar_function_from_sql_statement_postgres_syntax() -> Result<(
quote_style: None,
span: Span::empty(),
}),
data_type: DataType::Utf8,
data_type: DataType::Utf8View,
default_expr: None,
}]),
return_type: Some(DataType::Int32),
Expand Down
5 changes: 3 additions & 2 deletions datafusion/expr-common/src/type_coercion/binary.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -462,7 +462,7 @@ pub fn type_union_resolution(data_types: &[DataType]) -> Option<DataType> {

// If all the data_types are null, return string
if data_types.iter().all(|t| t == &DataType::Null) {
return Some(DataType::Utf8);
return Some(DataType::Utf8View);
}

// Ignore Nulls, if any data_type category is not the same, return None
Expand Down Expand Up @@ -1202,7 +1202,8 @@ pub fn string_coercion(lhs_type: &DataType, rhs_type: &DataType) -> Option<DataT
fn numeric_string_coercion(lhs_type: &DataType, rhs_type: &DataType) -> Option<DataType> {
use arrow::datatypes::DataType::*;
match (lhs_type, rhs_type) {
(Utf8 | LargeUtf8, other_type) | (other_type, Utf8 | LargeUtf8)
(Utf8 | LargeUtf8 | Utf8View, other_type)
| (other_type, Utf8 | LargeUtf8 | Utf8View)
if other_type.is_numeric() =>
{
Some(other_type.clone())
Expand Down
2 changes: 2 additions & 0 deletions datafusion/physical-plan/src/joins/sort_merge_join.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2487,6 +2487,7 @@ fn compare_join_arrays(
DataType::Float32 => compare_value!(Float32Array),
DataType::Float64 => compare_value!(Float64Array),
DataType::Utf8 => compare_value!(StringArray),
DataType::Utf8View => compare_value!(StringViewArray),
DataType::LargeUtf8 => compare_value!(LargeStringArray),
DataType::Decimal128(..) => compare_value!(Decimal128Array),
DataType::Timestamp(time_unit, None) => match time_unit {
Expand Down Expand Up @@ -2554,6 +2555,7 @@ fn is_join_arrays_equal(
DataType::Float32 => compare_value!(Float32Array),
DataType::Float64 => compare_value!(Float64Array),
DataType::Utf8 => compare_value!(StringArray),
DataType::Utf8View => compare_value!(StringViewArray),
DataType::LargeUtf8 => compare_value!(LargeStringArray),
DataType::Decimal128(..) => compare_value!(Decimal128Array),
DataType::Timestamp(time_unit, None) => match time_unit {
Expand Down
10 changes: 9 additions & 1 deletion datafusion/sql/src/planner.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ pub struct ParserOptions {
pub collect_spans: bool,
/// Whether `VARCHAR` is mapped to `Utf8View` during SQL planning.
pub map_varchar_to_utf8view: bool,
/// Whether `CHAR` and `Text` and `String` are mapped to `Utf8View` during SQL planning.
pub map_char_to_utf8view: bool,
}

impl ParserOptions {
Expand All @@ -73,6 +75,7 @@ impl ParserOptions {
enable_ident_normalization: true,
support_varchar_with_length: true,
map_varchar_to_utf8view: true,
map_char_to_utf8view: true,
enable_options_value_normalization: false,
collect_spans: false,
}
Expand Down Expand Up @@ -144,6 +147,7 @@ impl From<&SqlParserOptions> for ParserOptions {
enable_ident_normalization: options.enable_ident_normalization,
support_varchar_with_length: options.support_varchar_with_length,
map_varchar_to_utf8view: options.map_varchar_to_utf8view,
map_char_to_utf8view: options.map_char_to_utf8view,
enable_options_value_normalization: options
.enable_options_value_normalization,
collect_spans: options.collect_spans,
Expand Down Expand Up @@ -601,7 +605,11 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
)
}
SQLDataType::Char(_) | SQLDataType::Text | SQLDataType::String(_) => {
Ok(DataType::Utf8)
if self.options.map_char_to_utf8view {
Ok(DataType::Utf8View)
} else {
Ok(DataType::Utf8)
}
}
SQLDataType::Timestamp(precision, tz_info)
if precision.is_none() || [0, 3, 6, 9].contains(&precision.unwrap()) =>
Expand Down
12 changes: 6 additions & 6 deletions datafusion/sql/tests/cases/params.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -746,31 +746,31 @@ fn test_prepare_statement_to_plan_multi_params() {
assert_snapshot!(
plan,
@r#"
Prepare: "my_plan" [Int32, Utf8, Float64, Int32, Float64, Utf8]
Prepare: "my_plan" [Int32, Utf8View, Float64, Int32, Float64, Utf8View]
Projection: person.id, person.age, $6
Filter: person.age IN ([$1, $4]) AND person.salary > $3 AND person.salary < $5 OR person.first_name < $2
TableScan: person
"#
);
assert_snapshot!(dt, @r#"[Int32, Utf8, Float64, Int32, Float64, Utf8]"#);
assert_snapshot!(dt, @r#"[Int32, Utf8View, Float64, Int32, Float64, Utf8View]"#);

///////////////////
// replace params with values
let param_values = vec![
ScalarValue::Int32(Some(10)),
ScalarValue::from("abc"),
ScalarValue::Utf8View(Some("abc".into())),
ScalarValue::Float64(Some(100.0)),
ScalarValue::Int32(Some(20)),
ScalarValue::Float64(Some(200.0)),
ScalarValue::from("xyz"),
ScalarValue::Utf8View(Some("xyz".into())),
];

let plan_with_params = plan.with_param_values(param_values).unwrap();
assert_snapshot!(
plan_with_params,
@r#"
Projection: person.id, person.age, Utf8("xyz") AS $6
Filter: person.age IN ([Int32(10), Int32(20)]) AND person.salary > Float64(100) AND person.salary < Float64(200) OR person.first_name < Utf8("abc")
Projection: person.id, person.age, Utf8View("xyz") AS $6
Filter: person.age IN ([Int32(10), Int32(20)]) AND person.salary > Float64(100) AND person.salary < Float64(200) OR person.first_name < Utf8View("abc")
TableScan: person
"#
);
Expand Down
3 changes: 3 additions & 0 deletions datafusion/sql/tests/sql_integration.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3356,6 +3356,7 @@ fn parse_decimals_parser_options() -> ParserOptions {
enable_ident_normalization: false,
support_varchar_with_length: false,
map_varchar_to_utf8view: true,
map_char_to_utf8view: true,
enable_options_value_normalization: false,
collect_spans: false,
}
Expand All @@ -3367,6 +3368,7 @@ fn ident_normalization_parser_options_no_ident_normalization() -> ParserOptions
enable_ident_normalization: false,
support_varchar_with_length: false,
map_varchar_to_utf8view: true,
map_char_to_utf8view: true,
enable_options_value_normalization: false,
collect_spans: false,
}
Expand All @@ -3378,6 +3380,7 @@ fn ident_normalization_parser_options_ident_normalization() -> ParserOptions {
enable_ident_normalization: true,
support_varchar_with_length: false,
map_varchar_to_utf8view: true,
map_char_to_utf8view: true,
enable_options_value_normalization: false,
collect_spans: false,
}
Expand Down
8 changes: 4 additions & 4 deletions datafusion/sqllogictest/test_files/aggregate.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2278,10 +2278,10 @@ create table t (c string) as values
query T
select arrow_typeof(c) from t;
----
Utf8
Utf8
Utf8
Utf8
Utf8View
Utf8View
Utf8View
Utf8View

query IT
select count(c), arrow_typeof(count(c)) from t;
Expand Down
Loading