Skip to content

Commit 38c97b0

Browse files
Mike Artzasl3
authored andcommitted
[SPARK-51557][SQL][TESTS] Add tests for TIME data type in Java API
### What changes were proposed in this pull request? This PR adds tests for the TIME data type in Spark's Java API, covering Dataset and UDF functionality with `java.time.LocalTime`: - Adding a dataset filter operations with `java.time.LocalTime` (there is not a similar one for `TimestampType`). - UDF registration and execution with TimeType in udf8Test - the same as the `udf7Test()` for `TimestampType`. - `testLocalTimeEncoder()` already existed for `TimestampType` parity. ### Why are the changes needed? As part of the TIME data type SPIP (SPARK-51162), we need test coverage in the Java API for Datasets and UDFs. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added new test methods: - `JavaDatasetSuite.testLocalTimeFilter` - Tests Dataset filter with `LocalTime` - `JavaUDFSuite.udf8Test` - Tests UDF registration and execution with `LocalTime` ![Screenshot 2025-07-07 at 4 00 58 AM](https://github.com/user-attachments/assets/160f207e-e3e5-45a2-ac5d-35b55a76215e) ![Screenshot 2025-07-07 at 3 59 24 AM](https://github.com/user-attachments/assets/c18bf26d-a6ef-4445-95b0-a67566ac9fa0) I executed the tests themselves locally and also ran `./build/mvn test-compile -pl sql/core` to test compilation. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#51387 from fartzy/SPARK-51557_Add_tests_for_TIME_data_type_in_Java_API. Authored-by: Mike Artz <fartzy@hotmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>
1 parent 7a54cf6 commit 38c97b0

File tree

2 files changed

+31
-0
lines changed

2 files changed

+31
-0
lines changed

sql/core/src/test/java/test/org/apache/spark/sql/JavaDatasetSuite.java

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -812,6 +812,26 @@ public void testLocalTimeEncoder() {
812812
Assertions.assertEquals(data, ds.collectAsList());
813813
}
814814

815+
@Test
816+
public void testLocalTimeFilter() {
817+
Encoder<LocalTime> encoder = Encoders.LOCALTIME();
818+
List<LocalTime> data = Arrays.asList(
819+
LocalTime.of(9, 30, 45),
820+
LocalTime.of(14, 10, 10),
821+
LocalTime.of(22, 10, 10)
822+
);
823+
Dataset<LocalTime> ds = spark.createDataset(data, encoder);
824+
825+
Dataset<LocalTime> filtered = ds.filter(
826+
(FilterFunction<LocalTime>) time -> time.isAfter(LocalTime.of(12, 0, 0))
827+
);
828+
List<LocalTime> expectedFiltered = Arrays.asList(
829+
LocalTime.of(14, 10, 10),
830+
LocalTime.of(22, 10, 10)
831+
);
832+
Assertions.assertEquals(expectedFiltered, filtered.collectAsList());
833+
}
834+
815835
public static class KryoSerializable {
816836
String value;
817837

sql/core/src/test/java/test/org/apache/spark/sql/JavaUDFSuite.java

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919

2020
import java.io.Serializable;
2121
import java.time.LocalDate;
22+
import java.time.LocalTime;
2223
import java.util.List;
2324

2425
import org.apache.spark.sql.catalyst.FunctionIdentifier;
@@ -34,6 +35,7 @@
3435
import org.apache.spark.sql.classic.SparkSession;
3536
import org.apache.spark.sql.api.java.UDF2;
3637
import org.apache.spark.sql.types.DataTypes;
38+
import org.apache.spark.sql.types.TimeType;
3739

3840
// The test suite itself is Serializable so that anonymous Function implementations can be
3941
// serialized, as an alternative to converting these anonymous classes to static inner classes;
@@ -136,6 +138,15 @@ public void udf7Test() {
136138
}
137139
}
138140

141+
@Test
142+
public void udf8Test() {
143+
spark.udf().register(
144+
"plusTwoHours",
145+
(java.time.LocalTime lt) -> lt.plusHours(2), new TimeType(6));
146+
Row result = spark.sql("SELECT plusTwoHours(TIME '09:10:10')").head();
147+
Assertions.assertEquals(LocalTime.of(11, 10, 10), result.get(0));
148+
}
149+
139150
@Test
140151
public void sourceTest() {
141152
spark.udf().register("stringLengthTest", (String str) -> str.length(), DataTypes.IntegerType);

0 commit comments

Comments
 (0)