Skip to content

Commit f38f654

Browse files
authored
Merge pull request #269 from marklogic/feature/bump-spark
Testing against Spark 3.4.3
2 parents b28c471 + 341acc8 commit f38f654

File tree

4 files changed

+7
-14
lines changed

4 files changed

+7
-14
lines changed

CONTRIBUTING.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -176,8 +176,8 @@ spark.read.option("header", True).csv("src/test/resources/data.csv")\
176176
When you run PySpark, it will create its own Spark cluster. If you'd like to try against a separate Spark cluster
177177
that still runs on your local machine, perform the following steps:
178178

179-
1. Use [sdkman to install Spark](https://sdkman.io/sdks#spark). Run `sdk install spark 3.4.1` since we are currently
180-
building against Spark 3.4.1.
179+
1. Use [sdkman to install Spark](https://sdkman.io/sdks#spark). Run `sdk install spark 3.4.3` since we are currently
180+
building against Spark 3.4.3.
181181
2. `cd ~/.sdkman/candidates/spark/current/sbin`, which is where sdkman will install Spark.
182182
3. Run `./start-master.sh` to start a master Spark node.
183183
4. `cd ../logs` and open the master log file that was created to find the address for the master node. It will be in a

examples/entity-aggregation/build.gradle

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ repositories {
77
}
88

99
dependencies {
10-
implementation 'org.apache.spark:spark-sql_2.12:3.4.1'
10+
implementation 'org.apache.spark:spark-sql_2.12:3.4.3'
1111
implementation "com.marklogic:marklogic-spark-connector:2.2.0"
1212
implementation "org.postgresql:postgresql:42.6.2"
1313
}

examples/java-dependency/build.gradle

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ repositories {
77
}
88

99
dependencies {
10-
implementation 'org.apache.spark:spark-sql_2.12:3.4.1'
10+
implementation 'org.apache.spark:spark-sql_2.12:3.4.3'
1111
implementation 'com.marklogic:marklogic-spark-connector:2.2.0'
1212
}
1313

gradle.properties

Lines changed: 3 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,6 @@
1-
# Testing against 3.3.2 for the 2.0.0 release as 3.3.0 was released in June 2022 and 3.3.2 in February 2023, while
2-
# 3.4.0 is fairly new - April 2023. And at least AWS Glue and EMR are only on 3.3.0. But 3.3.2 has bug fixes that
3-
# affect some of our tests - see PushDownGroupByCountTest for an example. So we're choosing to build and test
4-
# against the latest 3.3.x release so we're not writing assertions based on buggy behavior in Spark 3.3.0.
5-
#
6-
# For 2.1.0, planning on using at least 3.4.x, and possibly 3.5.x. All tests are passing with 3.4.x when authors are
7-
# in a single document on MarkLogic 11. The tests that verify the number of rows read from MarkLogic (as opposed to
8-
# rows in the Spark dataset) will fail on MarkLogic 12 for now given that all rows come from the same document, and thus
9-
# all come from a single call to MarkLogic.
10-
sparkVersion=3.4.1
1+
# Staying with 3.4.x for now, as some pushdown tests are failing when using 3.5.x.
2+
# 3.4.3 release notes - https://spark.apache.org/releases/spark-release-3-4-3.html .
3+
sparkVersion=3.4.3
114

125
# Only used for the test app and for running tests.
136
mlHost=localhost

0 commit comments

Comments
 (0)