Add contains function, and support in datafusion substrait consumer #10879

Lordworms · 2024-06-11T21:19:32Z

Which issue does this PR close?

Closes #10861

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

waynexia · 2024-06-12T03:31:48Z

datafusion/functions/src/string/contains.rs

+) -> Result<ArrayRef, DataFusionError> {
+    let mod_str = as_generic_string_array::<T>(&args[0])?;
+    let match_str = as_generic_string_array::<T>(&args[1])?;
+    let res = arrow::compute::kernels::comparison::regexp_is_match_utf8(


Do we need to escape search string as it's used in regexp? Wondering what's the result of contains("abcdefg", ".*")

@Lordworms can you possibly respond to @waynexia 's comment?

To keep the review queue clear, I filed #10929 to follow up on this item and will merge this PR in

waynexia

Thanks @Lordworms, this looks good to me in general

alamb

Can we also please add an entry to the documentation about this new function? https://datafusion.apache.org/user-guide/sql/scalar_functions.html#scalar-functions

alamb · 2024-06-12T16:36:55Z

datafusion/functions/src/string/mod.rs

@@ -149,6 +149,9 @@ pub mod expr_fn {
    ),(
        uuid,
        "returns uuid v4 as a string value",
+    ), (
+        contains,
+        "Return true if search_string is found within string.",


I think we should also mention here that search_string is treated like a regex if that is the intention

datafusion/functions/src/string/contains.rs

Lordworms · 2024-06-13T02:23:25Z

Can we also please add an entry to the documentation about this new function? https://datafusion.apache.org/user-guide/sql/scalar_functions.html#scalar-functions

Sure, sorry for the ignorance

docs/source/user-guide/sql/scalar_functions.md

Co-authored-by: Alex Huang <huangweijun1001@gmail.com>

Weijun-H · 2024-06-14T04:24:57Z

Please add an entry here

datafusion/docs/source/user-guide/sql/scalar_functions.md

Lines 645 to 646 in 1a2a1bf

	## String Functions

alamb · 2024-06-14T20:31:26Z

The CI failure seems unrelated to the code in this PR: https://github.com/apache/datafusion/actions/runs/9510703358?pr=10879

alamb · 2024-06-15T16:45:53Z

Thanks again @waynexia @Lordworms and @Weijun-H !

…pache#10879) * adding new function contains * adding substrait test * adding doc * adding doc * Update docs/source/user-guide/sql/scalar_functions.md Co-authored-by: Alex Huang <huangweijun1001@gmail.com> * adding entry --------- Co-authored-by: Alex Huang <huangweijun1001@gmail.com>

…11060) * wip create and register ext file types with session * Add contains function, and support in datafusion substrait consumer (#10879) * adding new function contains * adding substrait test * adding doc * adding doc * Update docs/source/user-guide/sql/scalar_functions.md Co-authored-by: Alex Huang <huangweijun1001@gmail.com> * adding entry --------- Co-authored-by: Alex Huang <huangweijun1001@gmail.com> * logical planning updated * compiling * removing filetype enum * compiling * working on tests * fix some tests * test fixes * cli fix * cli fmt * Update datafusion/core/src/datasource/file_format/mod.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update datafusion/core/src/execution/session_state.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * review comments * review comments * review comments * typo fix * fmt * fix err log style * fmt --------- Co-authored-by: Lordworms <48054792+Lordworms@users.noreply.github.com> Co-authored-by: Alex Huang <huangweijun1001@gmail.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…pache#11060) * wip create and register ext file types with session * Add contains function, and support in datafusion substrait consumer (apache#10879) * adding new function contains * adding substrait test * adding doc * adding doc * Update docs/source/user-guide/sql/scalar_functions.md Co-authored-by: Alex Huang <huangweijun1001@gmail.com> * adding entry --------- Co-authored-by: Alex Huang <huangweijun1001@gmail.com> * logical planning updated * compiling * removing filetype enum * compiling * working on tests * fix some tests * test fixes * cli fix * cli fmt * Update datafusion/core/src/datasource/file_format/mod.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update datafusion/core/src/execution/session_state.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * review comments * review comments * review comments * typo fix * fmt * fix err log style * fmt --------- Co-authored-by: Lordworms <48054792+Lordworms@users.noreply.github.com> Co-authored-by: Alex Huang <huangweijun1001@gmail.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…pache#10879) * adding new function contains * adding substrait test * adding doc * adding doc * Update docs/source/user-guide/sql/scalar_functions.md Co-authored-by: Alex Huang <huangweijun1001@gmail.com> * adding entry --------- Co-authored-by: Alex Huang <huangweijun1001@gmail.com>

…pache#11060) * wip create and register ext file types with session * Add contains function, and support in datafusion substrait consumer (apache#10879) * adding new function contains * adding substrait test * adding doc * adding doc * Update docs/source/user-guide/sql/scalar_functions.md Co-authored-by: Alex Huang <huangweijun1001@gmail.com> * adding entry --------- Co-authored-by: Alex Huang <huangweijun1001@gmail.com> * logical planning updated * compiling * removing filetype enum * compiling * working on tests * fix some tests * test fixes * cli fix * cli fmt * Update datafusion/core/src/datasource/file_format/mod.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update datafusion/core/src/execution/session_state.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * review comments * review comments * review comments * typo fix * fmt * fix err log style * fmt --------- Co-authored-by: Lordworms <48054792+Lordworms@users.noreply.github.com> Co-authored-by: Alex Huang <huangweijun1001@gmail.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Lordworms added 2 commits June 11, 2024 13:56

adding new function contains

fc7aa35

adding substrait test

41986e5

github-actions bot added sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate labels Jun 11, 2024

Lordworms marked this pull request as ready for review June 11, 2024 23:35

waynexia reviewed Jun 12, 2024

View reviewed changes

waynexia approved these changes Jun 12, 2024

View reviewed changes

alamb changed the title ~~Support for contains function in datafusion substrait consumer~~ Add contains function, and support in in datafusion substrait consumer Jun 12, 2024

alamb changed the title ~~Add contains function, and support in in datafusion substrait consumer~~ Add contains function, and support in datafusion substrait consumer Jun 12, 2024

alamb reviewed Jun 12, 2024

View reviewed changes

Lordworms added 2 commits June 12, 2024 19:52

adding doc

a85012a

adding doc

1a2a1bf

Weijun-H reviewed Jun 14, 2024

View reviewed changes

docs/source/user-guide/sql/scalar_functions.md Outdated Show resolved Hide resolved

Update docs/source/user-guide/sql/scalar_functions.md

3ff603a

Co-authored-by: Alex Huang <huangweijun1001@gmail.com>

adding entry

2cf62a0

alamb mentioned this pull request Jun 15, 2024

Do we need to escape search string as it's used in regexp? Wondering what's the result of contains("abcdefg", ".*") #10929

Open

alamb merged commit 87aea14 into apache:main Jun 15, 2024
24 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add contains function, and support in datafusion substrait consumer #10879

Add contains function, and support in datafusion substrait consumer #10879

Uh oh!

Lordworms commented Jun 11, 2024

Uh oh!

waynexia Jun 12, 2024 •

edited

Loading

Uh oh!

alamb Jun 13, 2024

Uh oh!

alamb Jun 15, 2024

Uh oh!

waynexia left a comment

Uh oh!

alamb left a comment

Uh oh!

alamb Jun 12, 2024 •

edited

Loading

Uh oh!

Lordworms Jun 14, 2024

Uh oh!

Uh oh!

Lordworms commented Jun 13, 2024 •

edited

Loading

Uh oh!

Uh oh!

Weijun-H commented Jun 14, 2024

Uh oh!

alamb commented Jun 14, 2024

Uh oh!

alamb commented Jun 15, 2024

Uh oh!

Uh oh!

Uh oh!

Add contains function, and support in datafusion substrait consumer #10879

Add contains function, and support in datafusion substrait consumer #10879

Uh oh!

Conversation

Lordworms commented Jun 11, 2024

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

waynexia Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb Jun 13, 2024

Choose a reason for hiding this comment

Uh oh!

alamb Jun 15, 2024

Choose a reason for hiding this comment

Uh oh!

waynexia left a comment

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lordworms Jun 14, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Lordworms commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Weijun-H commented Jun 14, 2024

Uh oh!

alamb commented Jun 14, 2024

Uh oh!

alamb commented Jun 15, 2024

Uh oh!

Uh oh!

Uh oh!

waynexia Jun 12, 2024 •

edited

Loading

alamb Jun 12, 2024 •

edited

Loading

Lordworms commented Jun 13, 2024 •

edited

Loading