Testing against Spark 3.4.3

3.4.3 is the latest 3.4.x release and is "strongly recommended" by the Spark team. This change does not impact our connector though, as we don't include any Spark libraries in our connector. It's just ensuring we compile and test against the latest 3.4.x release.

Author: rjrudin

CONTRIBUTING.md

@@ -176,8 +176,8 @@ spark.read.option("header", True).csv("src/test/resources/data.csv")\
 When you run PySpark, it will create its own Spark cluster. If you'd like to try against a separate Spark cluster
 that still runs on your local machine, perform the following steps:
 
-1. Use [sdkman to install Spark](https://sdkman.io/sdks#spark). Run `sdk install spark 3.4.1` since we are currently
-building against Spark 3.4.1.
+1. Use [sdkman to install Spark](https://sdkman.io/sdks#spark). Run `sdk install spark 3.4.3` since we are currently
+building against Spark 3.4.3.
 2. `cd ~/.sdkman/candidates/spark/current/sbin`, which is where sdkman will install Spark.
 3. Run `./start-master.sh` to start a master Spark node.
 4. `cd ../logs` and open the master log file that was created to find the address for the master node. It will be in a

examples/entity-aggregation/build.gradle

@@ -7,7 +7,7 @@ repositories {
 }
 
 dependencies {
-  implementation 'org.apache.spark:spark-sql_2.12:3.4.1'
+  implementation 'org.apache.spark:spark-sql_2.12:3.4.3'
   implementation "com.marklogic:marklogic-spark-connector:2.2.0"
   implementation "org.postgresql:postgresql:42.6.2"
 }

examples/java-dependency/build.gradle

@@ -7,7 +7,7 @@ repositories {
 }
 
 dependencies {
-  implementation 'org.apache.spark:spark-sql_2.12:3.4.1'
+  implementation 'org.apache.spark:spark-sql_2.12:3.4.3'
   implementation 'com.marklogic:marklogic-spark-connector:2.2.0'
 }
 

gradle.properties

@@ -1,13 +1,6 @@
-# Testing against 3.3.2 for the 2.0.0 release as 3.3.0 was released in June 2022 and 3.3.2 in February 2023, while
-# 3.4.0 is fairly new - April 2023. And at least AWS Glue and EMR are only on 3.3.0. But 3.3.2 has bug fixes that
-# affect some of our tests - see PushDownGroupByCountTest for an example. So we're choosing to build and test
-# against the latest 3.3.x release so we're not writing assertions based on buggy behavior in Spark 3.3.0.
-#
-# For 2.1.0, planning on using at least 3.4.x, and possibly 3.5.x. All tests are passing with 3.4.x when authors are
-# in a single document on MarkLogic 11. The tests that verify the number of rows read from MarkLogic (as opposed to
-# rows in the Spark dataset) will fail on MarkLogic 12 for now given that all rows come from the same document, and thus
-# all come from a single call to MarkLogic.
-sparkVersion=3.4.1
+# Staying with 3.4.x for now, as some pushdown tests are failing when using 3.5.x.
+# 3.4.3 release notes - https://spark.apache.org/releases/spark-release-3-4-3.html .
+sparkVersion=3.4.3
 
 # Only used for the test app and for running tests.
 mlHost=localhost