[SPARK-52617][SQL] Cast TIME to/from TIMESTAMP_NTZ #51381

SubhamSinghal · 2025-07-05T11:13:37Z

What changes were proposed in this pull request?

This PR adds support for casting TIME to/from TIMESTAMP_NTZ type

Why are the changes needed?

Adds casting support between 2 types.

Does this PR introduce any user-facing change?

yes

How was this patch tested?

By running the related test suites: CastSuiteBase

By manual tests:

timestamp_ntz to time conversion:

scala> spark.sql(
     |   """
     |     |SELECT CAST(CAST('2025-07-05 12:34:56.123456' AS TIMESTAMP_NTZ) AS TIME(6))
     |   """.stripMargin
     | ).show(false)

+------------------------------------------------------------------+
|CAST(CAST(2025-07-05 12:34:56.123456 AS TIMESTAMP_NTZ) AS TIME(6))|
+------------------------------------------------------------------+
|12:34:56.123456                                                   |
+------------------------------------------------------------------+

time to timestamp_ntz conversion:

scala> spark.sql("""
     |   SELECT CAST(CAST('12:34:56.123456' AS TIME(6)) AS TIMESTAMP_NTZ)
     | """).show(false)
+-------------------------------------------------------+
|CAST(CAST(12:34:56.123456 AS TIME(6)) AS TIMESTAMP_NTZ)|
+-------------------------------------------------------+
|2025-07-05 12:34:56.123456                             |
+-------------------------------------------------------+

logical plan analysis:

scala> spark.sql("SELECT CAST(make_time(12, 30, 40) AS TIMESTAMP_NTZ)").explain(true)

== Analyzed Logical Plan ==
make_timestamp_ntz(current_date(), make_time(12, 30, 40)): timestamp_ntz
Project [make_timestamp_ntz(current_date(Some(Asia/Kolkata)), make_time(12, 30, cast(40 as decimal(16,6)))) AS make_timestamp_ntz(current_date(), make_time(12, 30, 40))#0]
+- OneRowRelation

Was this patch authored or co-authored using generative AI tooling?

yes
Generated / corrected with AI assistance: chatGPT

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

...st/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteTimeCastToTimestampNTZ.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala

...in/scala/org/apache/spark/sql/catalyst/analysis/resolver/RewriteTimeCastToTimestampNTZ.scala

.../scala/org/apache/spark/sql/catalyst/analysis/resolver/TimezoneAwareExpressionResolver.scala

…ysis/RewriteTimeCastToTimestampNTZ.scala Co-authored-by: Mihailo Timotic <mihailo.timotic@databricks.com>

…/DateTimeUtils.scala Co-authored-by: Mihailo Timotic <mihailo.timotic@databricks.com>

subham611 · 2025-07-10T12:49:41Z

On a separate note, is the new rule necessary? Should we try and place this in existing type coercion rules?

@mihailotim-db Should I make RewriteTimeCastToTimestampNTZ extend TypeCoercionRule instead of Rule[LogicalPlan] and place RewriteTimeCastToTimestampNTZ inside CombinedTypeCoercionRule?

.../scala/org/apache/spark/sql/catalyst/analysis/resolver/TimezoneAwareExpressionResolver.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala

MaxGekk · 2025-07-16T09:24:33Z

@SubhamSinghal Could you resolve conflicts, please.

mihailotim-db · 2025-07-16T09:43:00Z

.../scala/org/apache/spark/sql/catalyst/analysis/resolver/TimezoneAwareExpressionResolver.scala

+      case Cast(child, TimestampNTZType, _, _) if child.dataType.isInstanceOf[TimeType] =>
+        MakeTimestampNTZ(CurrentDate(), child)


There is no need to have this if we have a type coercion rule. Type coercion rules run implicitly in single-pass

@mihailotim-db are you suggesting to revert all change in TimeZoneAwareExpressionResolver class and type coercion rule will take care of single pass and fixed point analyzer?

mihailotim-db · 2025-07-16T09:44:17Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercion.scala

@@ -93,6 +93,7 @@ object AnsiTypeCoercion extends TypeCoercionBase {
      StackCoercion ::
      Division ::
      IntegralDivision ::
+      RewriteTimeCastToTimestampNTZ ::


Why is this a separate rule? I think we should be able to make it part of one of the existing rule, maybe DateTimeOperations @MaxGekk ?

Actually is this Cast coming from internal type coercion or user specified casts? If former, we should perform this change in the place where we add a cast. If latter, we should have an analysis rule for rewrite and not a type coercion one, since there is no coercion going on here

@MaxGekk Can you help here?

subham611 added 3 commits July 5, 2025 15:36

[SPARK-52617][SQL]Cast TIME to/from TIMESTAMP_NTZ

8799de5

Adds time to TimestampNTZType conversion

c37bbf0

Fix linting

352aacb

github-actions bot added the SQL label Jul 5, 2025

SubhamSinghal changed the title ~~[SPARK-52617][SQL]Cast TIME to/from TIMESTAMP_NTZ~~ [WIP][SPARK-52617][SQL]Cast TIME to/from TIMESTAMP_NTZ Jul 5, 2025

subham611 added 6 commits July 5, 2025 17:10

Remove unused import

7faed87

Adds can cast

11c3a26

Fix import

4be2758

Resolved ambiguous import

e020fae

Fix UT failure

5b9a10e

Enable casting in ansi mode

d6f4424

MaxGekk requested changes Jul 5, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala Outdated Show resolved Hide resolved

SubhamSinghal changed the title ~~[WIP][SPARK-52617][SQL]Cast TIME to/from TIMESTAMP_NTZ~~ [SPARK-52617][SQL]Cast TIME to/from TIMESTAMP_NTZ Jul 5, 2025

subham611 added 2 commits July 5, 2025 22:25

Calculate current day outside buildCast

f60c6cb

Adds RewriteTimeCastToTimestampNTZ rule

828d214

SubhamSinghal force-pushed the SPARK-52617_cast_time_to/from_TIMESTAMP_NTZ branch from 4167bf1 to 828d214 Compare July 6, 2025 16:05

subham611 added 6 commits July 6, 2025 21:39

Fix additional change

90482d6

Fix unused import

a9d1bdd

Fix UT

fdc9b92

Fix UT failure

b919e4e

Fix UT

e2ef8af

Fix UT

0bada28

MaxGekk reviewed Jul 7, 2025

View reviewed changes

subham611 added 2 commits July 7, 2025 18:09

Move to resolver

2a5e9ad

Revert unwanted changes

7d77c1f

mihailotim-db reviewed Jul 7, 2025

View reviewed changes

...in/scala/org/apache/spark/sql/catalyst/analysis/resolver/RewriteTimeCastToTimestampNTZ.scala Outdated Show resolved Hide resolved

.../scala/org/apache/spark/sql/catalyst/analysis/resolver/TimezoneAwareExpressionResolver.scala Outdated Show resolved Hide resolved

subham611 added 4 commits July 7, 2025 18:40

Resolve comment

cb6ad55

Delete resolver

cd503a5

Fix import

8bc58da

Add back RewriteTimeCastToTimestampNTZ rule

8a30b2a

SubhamSinghal force-pushed the SPARK-52617_cast_time_to/from_TIMESTAMP_NTZ branch from fa1daa9 to fd1aef3 Compare July 8, 2025 07:42

subham611 and others added 13 commits July 8, 2025 13:22

Resolve comments

5c6c97f

Update sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/anal…

6169ea4

…ysis/RewriteTimeCastToTimestampNTZ.scala Co-authored-by: Mihailo Timotic <mihailo.timotic@databricks.com>

Update sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util…

29ca860

…/DateTimeUtils.scala Co-authored-by: Mihailo Timotic <mihailo.timotic@databricks.com>

Merge master

011b254

Resolve comments

ca02825

remove new line

72bb9b5

Fix lint

68f0c98

Lint fix

6b5ed62

Modify rule

e53bfe3

Rewrite rule

b488aae

Fix import order

867967e

Adds codegen

ebf16ed

Fix UT

58c0bea

SubhamSinghal requested a review from MaxGekk July 9, 2025 11:25

subham611 added 2 commits July 10, 2025 10:16

Resolve comment

b82e21c

Lint fix

4239bef

Adds type coersion rule

fd5e76e

SubhamSinghal force-pushed the SPARK-52617_cast_time_to/from_TIMESTAMP_NTZ branch from f7d1226 to fd5e76e Compare July 10, 2025 13:51

SubhamSinghal requested a review from mihailotim-db July 10, 2025 15:59

MaxGekk changed the title ~~[SPARK-52617][SQL]Cast TIME to/from TIMESTAMP_NTZ~~ [SPARK-52617][SQL] Cast TIME to/from TIMESTAMP_NTZ Jul 15, 2025

MaxGekk requested changes Jul 15, 2025

View reviewed changes

.../scala/org/apache/spark/sql/catalyst/analysis/resolver/TimezoneAwareExpressionResolver.scala Show resolved Hide resolved

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala Outdated Show resolved Hide resolved

mihailotim-db suggested changes Jul 16, 2025

View reviewed changes

mihailotim-db reviewed Jul 16, 2025

View reviewed changes

subham611 added 2 commits July 16, 2025 16:02

Resolve merge conflict

89dfef0

Update nanos in a day function

68dc4a9

SubhamSinghal force-pushed the SPARK-52617_cast_time_to/from_TIMESTAMP_NTZ branch from e15a217 to 68dc4a9 Compare July 16, 2025 10:34

Merge master

76abdaf

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-52617][SQL] Cast TIME to/from TIMESTAMP_NTZ #51381

[SPARK-52617][SQL] Cast TIME to/from TIMESTAMP_NTZ #51381

Uh oh!

SubhamSinghal commented Jul 5, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

subham611 commented Jul 10, 2025

Uh oh!

Uh oh!

Uh oh!

MaxGekk commented Jul 16, 2025

Uh oh!

mihailotim-db Jul 16, 2025

Uh oh!

SubhamSinghal Jul 16, 2025

Uh oh!

mihailotim-db Jul 16, 2025

Uh oh!

mihailotim-db Jul 16, 2025

Uh oh!

SubhamSinghal Jul 21, 2025

Uh oh!

Uh oh!

		case Cast(child, TimestampNTZType, _, _) if child.dataType.isInstanceOf[TimeType] =>
		MakeTimestampNTZ(CurrentDate(), child)

[SPARK-52617][SQL] Cast TIME to/from TIMESTAMP_NTZ #51381

Are you sure you want to change the base?

[SPARK-52617][SQL] Cast TIME to/from TIMESTAMP_NTZ #51381

Uh oh!

Conversation

SubhamSinghal commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

subham611 commented Jul 10, 2025

Uh oh!

Uh oh!

Uh oh!

MaxGekk commented Jul 16, 2025

Uh oh!

mihailotim-db Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

SubhamSinghal Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

mihailotim-db Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

mihailotim-db Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

SubhamSinghal Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SubhamSinghal commented Jul 5, 2025 •

edited

Loading