Skip to content

Commit 25b0f7b

Browse files
bersprocketsMaxGekk
authored andcommitted
[SPARK-52699][SQL] Support aggregating TIME type in interpreted mode
### What changes were proposed in this pull request? Update `InternalRow#getWriter` to return the correct writer for the TIME type. ### Why are the changes needed? Without this PR, aggregating the TIME type in interpreted mode fails. Consider the below query: ``` set spark.sql.codegen.factoryMode=NO_CODEGEN; create or replace temp view v1(col1) as values (time'22:33:01'), (time'01:33:01'), (null); select max(col1), min(col1) from v1; ``` Without this change, the query fails with the following error: ``` Exception in task 0.0 in stage 0.0 (TID 0) org.apache.spark.SparkUnsupportedOperationException: [UNSUPPORTED_CALL.WITHOUT_SUGGESTION] Cannot call the method "update" of the class "org.apache.spark.sql.catalyst.expressions.UnsafeRow". SQLSTATE: 0A000 at org.apache.spark.SparkUnsupportedOperationException$.apply(SparkException.scala:266) ~[spark-common-utils_2.13-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] ``` ### Does this PR introduce _any_ user-facing change? No. The TIME type is not released yet. ### How was this patch tested? Updated unit test. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #51395 from bersprockets/time_get_writer. Authored-by: Bruce Robbins <bersprockets@gmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>
1 parent 62becf4 commit 25b0f7b

File tree

2 files changed

+5
-4
lines changed

2 files changed

+5
-4
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -175,7 +175,7 @@ object InternalRow {
175175
case ShortType => (input, v) => input.setShort(ordinal, v.asInstanceOf[Short])
176176
case IntegerType | DateType | _: YearMonthIntervalType =>
177177
(input, v) => input.setInt(ordinal, v.asInstanceOf[Int])
178-
case LongType | TimestampType | TimestampNTZType | _: DayTimeIntervalType =>
178+
case LongType | TimestampType | TimestampNTZType | _: DayTimeIntervalType | _: TimeType =>
179179
(input, v) => input.setLong(ordinal, v.asInstanceOf[Long])
180180
case FloatType => (input, v) => input.setFloat(ordinal, v.asInstanceOf[Float])
181181
case DoubleType => (input, v) => input.setDouble(ordinal, v.asInstanceOf[Double])

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MutableProjectionSuite.scala

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,8 @@ class MutableProjectionSuite extends SparkFunSuite with ExpressionEvalHelper {
3232

3333
val fixedLengthTypes = Array[DataType](
3434
BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType,
35-
DateType, TimestampType) ++ dayTimeIntervalTypes ++ yearMonthIntervalTypes
35+
DateType, TimestampType, TimestampNTZType, new TimeType(6)) ++
36+
dayTimeIntervalTypes ++ yearMonthIntervalTypes
3637

3738
val variableLengthTypes = Array(
3839
StringType, DecimalType.defaultConcreteType, CalendarIntervalType, BinaryType,
@@ -46,7 +47,7 @@ class MutableProjectionSuite extends SparkFunSuite with ExpressionEvalHelper {
4647

4748
testBothCodegenAndInterpreted("fixed-length types") {
4849
val inputRow = InternalRow.fromSeq(Seq(
49-
true, 3.toByte, 15.toShort, -83, 129L, 1.0f, 5.0, 1, 2L) ++
50+
true, 3.toByte, 15.toShort, -83, 129L, 1.0f, 5.0, 1, 2L, 3L, 4L) ++
5051
Seq.tabulate(dayTimeIntervalTypes.length)(_ => Long.MaxValue) ++
5152
Seq.tabulate(yearMonthIntervalTypes.length)(_ => Int.MaxValue))
5253
val proj = createMutableProjection(fixedLengthTypes)
@@ -55,7 +56,7 @@ class MutableProjectionSuite extends SparkFunSuite with ExpressionEvalHelper {
5556

5657
testBothCodegenAndInterpreted("unsafe buffer") {
5758
val inputRow = InternalRow.fromSeq(Seq(
58-
false, 1.toByte, 9.toShort, -18, 53L, 3.2f, 7.8, 4, 9L) ++
59+
false, 1.toByte, 9.toShort, -18, 53L, 3.2f, 7.8, 4, 9L, 10L, 11L) ++
5960
Seq.tabulate(dayTimeIntervalTypes.length)(_ => Long.MaxValue) ++
6061
Seq.tabulate(yearMonthIntervalTypes.length)(_ => Int.MaxValue))
6162
val numFields = fixedLengthTypes.length

0 commit comments

Comments
 (0)