MLE-23024 Revert prior PR #523
Conversation
rjrudin
changed the title
rjrudin
force-pushed
the
feature/23024-revert
branch
from
2c9f3ea to
30bce5b
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR reverts the Spark 4.0 upgrade by rolling back commit 678c21d, which introduced Spark 4.0 support. The revert returns the codebase to use Spark 3.5.6 instead of Spark 4.0.0.
- Downgrade Spark version from 4.0.0 to 3.5.6
- Update Scala collections API usage from Spark 4.0 patterns back to Spark 3.5-compatible patterns
- Remove Spark 4.0-specific workarounds and adjustments
Reviewed Changes
Copilot reviewed 18 out of 19 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| gradle.properties | Reverts Spark version from 4.0.0 to 3.5.6 |
| marklogic-spark-connector/build.gradle | Changes Spark SQL dependency from 2.13 to 2.12 |
| tests/build.gradle | Reverts Spark SQL test dependency and removes increased heap size JVM args |
| Multiple test files | Reverts Scala collection API usage from CollectionConverters back to JavaConversions/JavaConverters |
| Multiple connector files | Removes @NotNull annotations and reverts Scala API changes |
| AbstractIntegrationTest.java | Reverts from IOUtils.closeQuietly back to direct close() call |
| AbstractPushDownTest.java | Updates method name and logic from Spark 4.0 back to 3.5 handling |
| @@ -49,9 +49,8 @@ public InternalRow deserializeJson(String json) { | |||
|
|
|||
| private JacksonParser newJacksonParser(StructType schema) { | |||
| final boolean allowArraysAsStructs = true; | |||
| List<Filter> filters = new ArrayList<>(); | |||
| scala.collection.immutable.Seq<Filter> filterSeq = scala.collection.immutable.List.from(CollectionConverters.asScala(filters)).toSeq(); | |||
| final Seq<Filter> filters = JavaConverters.asScalaIterator(new ArrayList<Filter>().iterator()).toSeq(); | |||
There was a problem hiding this comment.
Creating a new ArrayList just to get an empty iterator is inefficient. Consider using Collections.emptyList().iterator() or a more direct approach to create an empty Scala Seq.
Copilot uses AI. Check for mistakes.
| @@ -12,8 +12,8 @@ | |||
| import org.junit.jupiter.api.AfterEach; | |||
| import org.junit.jupiter.api.Test; | |||
| import org.junit.jupiter.api.io.TempDir; | |||
| import scala.collection.Seq; | |||
| import scala.jdk.javaapi.CollectionConverters; | |||
| import scala.collection.JavaConversions; | |||
There was a problem hiding this comment.
JavaConversions is deprecated in newer Scala versions. While this may be necessary for Spark 3.5.6 compatibility, consider using JavaConverters when possible for better future-proofing.
| import scala.collection.JavaConversions; | |
| import scala.collection.JavaConverters; |
Copilot uses AI. Check for mistakes.
| @@ -16,8 +16,8 @@ | |||
| import org.junit.jupiter.api.BeforeEach; | |||
| import org.junit.jupiter.api.Test; | |||
| import org.junit.jupiter.api.io.TempDir; | |||
| import scala.collection.Seq; | |||
| import scala.jdk.javaapi.CollectionConverters; | |||
| import scala.collection.JavaConversions; | |||
There was a problem hiding this comment.
JavaConversions is deprecated in newer Scala versions. While this may be necessary for Spark 3.5.6 compatibility, consider using JavaConverters when possible for better future-proofing.
| import scala.collection.JavaConversions; | |
| import scala.collection.JavaConverters; |
Copilot uses AI. Check for mistakes.
Meant to merge it to feature/develop-4.0.0
rjrudin
force-pushed
the
feature/23024-revert
branch
from
30bce5b to
995f06f
Compare
rjrudin
changed the title
This reverts commit 678c21d, reversing
changes made to ba227c7.