Merge pull request #283 from marklogic/release/2.3.1

Merge release/2.3.1 into master

Author: rjrudin

.gitignore

@@ -16,5 +16,7 @@ logs
 .ipynb_checkpoints
 venv
 .venv
-docker
+docker/marklogic
+docker/sonarqube/data
+docker/sonarqube/logs
 export

CONTRIBUTING.md

@@ -83,6 +83,13 @@ you've introduced on the feature branch you're working on. You can then click on
 Note that if you only need results on code smells and vulnerabilities, you can repeatedly run `./gradlew sonar`
 without having to re-run the tests.
 
+Our Sonar instance is also configured to scan for dependency vulnerabilities 
+[via the dependency-check plugin](https://github.com/dependency-check/dependency-check-sonar-plugin). For more 
+information, see the `dependencyCheck` block in this project's `build.gradle` file. To include dependency check results,
+just run the following (it's not included by default when running the `sonar` task):
+
+    ./gradlew dependencyCheckAnalyze sonar
+
 ## Accessing MarkLogic logs in Grafana
 
 This project's `docker-compose-3nodes.yaml` file includes
@@ -116,7 +123,7 @@ This will produce a single jar file for the connector in the `./build/libs` dire
 
 You can then launch PySpark with the connector available via:
 
-    pyspark --jars build/libs/marklogic-spark-connector-2.3.0.jar
+    pyspark --jars build/libs/marklogic-spark-connector-2.3-SNAPSHOT.jar
 
 The below command is an example of loading data from the test application deployed via the instructions at the top of 
 this page. 
@@ -192,7 +199,7 @@ The Spark master GUI is at <http://localhost:8080>. You can use this to view det
 
 Now that you have a Spark cluster running, you just need to tell PySpark to connect to it:
 
-    pyspark --master spark://NYWHYC3G0W:7077 --jars build/libs/marklogic-spark-connector-2.3.0.jar
+    pyspark --master spark://NYWHYC3G0W:7077 --jars build/libs/marklogic-spark-connector-2.3-SNAPSHOT.jar
 
 You can then run the same commands as shown in the PySpark section above. The Spark master GUI will allow you to 
 examine details of each of the commands that you run.
@@ -211,12 +218,12 @@ You will need the connector jar available, so run `./gradlew clean shadowJar` if
 You can then run a test Python program in this repository via the following (again, change the master address as 
 needed); note that you run this outside of PySpark, and `spark-submit` is available after having installed PySpark:
 
-    spark-submit --master spark://NYWHYC3G0W:7077 --jars build/libs/marklogic-spark-connector-2.3.0.jar src/test/python/test_program.py
+    spark-submit --master spark://NYWHYC3G0W:7077 --jars build/libs/marklogic-spark-connector-2.3-SNAPSHOT.jar src/test/python/test_program.py
 
 You can also test a Java program. To do so, first move the `com.marklogic.spark.TestProgram` class from `src/test/java`
 to `src/main/java`. Then run `./gradlew clean shadowJar` to rebuild the connector jar. Then run the following:
 
-    spark-submit --master spark://NYWHYC3G0W:7077 --class com.marklogic.spark.TestProgram build/libs/marklogic-spark-connector-2.3.0.jar
+    spark-submit --master spark://NYWHYC3G0W:7077 --class com.marklogic.spark.TestProgram build/libs/marklogic-spark-connector-2.3-SNAPSHOT.jar
 
 Be sure to move `TestProgram` back to `src/test/java` when you are done. 
 

LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright © 2024 MarkLogic Corporation.
+Copyright © 2024 MarkLogic Corporation. All Rights Reserved.
 
 Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
 

NOTICE.txt

@@ -1,16 +1,12 @@
 MarkLogic® Connector for Spark
 
-Copyright © 2023 MarkLogic Corporation.  MarkLogic and MarkLogic logo are trademarks or registered trademarks of MarkLogic Corporation in the United States and other countries.  All other trademarks are the property of their respective owners.
- 
-This project is licensed under the Apache License, Version 2.0 (the "License"); you may not use this project except in compliance with the License. You may obtain a copy of the License at
+Copyright © 2024 MarkLogic Corporation. All Rights Reserved.
 
-http://www.apache.org/licenses/LICENSE-2.0 
-
-Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
-
-To the extent required by the applicable open-source license, a complete machine-readable copy of the source code corresponding to such code is available upon request. This offer is valid to anyone in receipt of this information and shall expire three years following the date of the final distribution of this product version by MarkLogic Corporation. To obtain such source code, send an email to Legal-thirdpartyreview@progress.com. Please specify the product and version for which you are requesting source code.
-
-The following software may be included in this project (last updated October 3, 2023):
+To the extent required by the applicable open-source license, a complete machine-readable copy of the source code
+corresponding to such code is available upon request. This offer is valid to anyone in receipt of this information and
+shall expire three years following the date of the final distribution of this product version by
+Progress Software Corporation. To obtain such source code, send an email to Legal-thirdpartyreview@progress.com.
+Please specify the product and version for which you are requesting source code.
 
 -------------------------------------------------------------------------
 Third Party Components
@@ -43,8 +39,8 @@ Apache License
 
    1. Definitions.
 
-      "License" shall mean the terms and conditions for use, 
-       reproduction, and distribution as defined by Sections 1 through 9 
+      "License" shall mean the terms and conditions for use,
+       reproduction, and distribution as defined by Sections 1 through 9
        of this document.
 
       "Licensor" shall mean the copyright owner or entity authorized by
@@ -75,20 +71,20 @@ Apache License
 
       "Derivative Works" shall mean any work, whether in Source or Object
       form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other 
-       modifications represent, as a whole, an original work of 
-       authorship. For the purposes of this License, Derivative Works 
-       shall not include works that remain separable from, or merely link 
-       (or bind by name) to the interfaces of, the Work and Derivative 
+      editorial revisions, annotations, elaborations, or other
+       modifications represent, as a whole, an original work of
+       authorship. For the purposes of this License, Derivative Works
+       shall not include works that remain separable from, or merely link
+       (or bind by name) to the interfaces of, the Work and Derivative
        Works thereof.
 
        "Contribution" shall mean any work of authorship, including the
        original version of the Work and any modifications or additions
       to that Work or Derivative Works thereof, that is intentionally
 submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
 
-      "Contributor" shall mean Licensor and any individual or Legal 
-       Entity on behalf of whom a Contribution has been received by 
+      "Contributor" shall mean Licensor and any individual or Legal
+       Entity on behalf of whom a Contribution has been received by
        Licensor and subsequently incorporated within the Work.
 
    2. Grant of Copyright License. Subject to the terms and conditions of
@@ -102,16 +98,16 @@ submitted to Licensor for inclusion in the Work by the copyright owner or by an
       this License, each Contributor hereby grants to You a perpetual,
       worldwide, non-exclusive, no-charge, royalty-free, irrevocable
       (except as stated in this section) patent license to make, have
-      made, use, offer to sell, sell, import, and otherwise transfer the 
-      Work, where such license applies only to those patent claims 
-      licensable by such Contributor that are necessarily infringed by 
-      their Contribution(s) alone or by combination of their 
-      Contribution(s) with the Work to which such Contribution(s) was 
-      submitted. If You institute patent litigation against any entity 
-      (including a cross-claim or counterclaim in a lawsuit) alleging 
-      that the Work or a Contribution incorporated within the Work 
-      constitutes direct or contributory patent infringement, then any 
-      patent licenses granted to You under this License for that Work 
+      made, use, offer to sell, sell, import, and otherwise transfer the
+      Work, where such license applies only to those patent claims
+      licensable by such Contributor that are necessarily infringed by
+      their Contribution(s) alone or by combination of their
+      Contribution(s) with the Work to which such Contribution(s) was
+      submitted. If You institute patent litigation against any entity
+      (including a cross-claim or counterclaim in a lawsuit) alleging
+      that the Work or a Contribution incorporated within the Work
+      constitutes direct or contributory patent infringement, then any
+      patent licenses granted to You under this License for that Work
       shall terminate as of the date such litigation is filed.
 
    4. Redistribution. You may reproduce and distribute copies of the
@@ -173,11 +169,11 @@ submitted to Licensor for inclusion in the Work by the copyright owner or by an
       agreed to in writing, Licensor provides the Work (and each
       Contributor provides its Contributions) on an "AS IS" BASIS,
       WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or 
-      conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS 
-      FOR A PARTICULAR PURPOSE. You are solely responsible for 
-      determining the appropriateness of using or redistributing the Work 
-      and assume any risks associated with Your exercise of permissions 
+      implied, including, without limitation, any warranties or
+      conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS
+      FOR A PARTICULAR PURPOSE. You are solely responsible for
+      determining the appropriateness of using or redistributing the Work
+      and assume any risks associated with Your exercise of permissions
       under this License.
 
    8. Limitation of Liability. In no event and under no legal theory,

build.gradle

@@ -7,10 +7,11 @@ plugins {
   id 'signing'
   id "jacoco"
   id "org.sonarqube" version "4.4.1.3373"
+  id "org.owasp.dependencycheck" version "10.0.3"
 }
 
 group 'com.marklogic'
-version '2.3.0'
+version '2.3.1'
 
 java {
   // To support reading RDF files, Apache Jena is used - but that requires Java 11. If we want to do a 2.2.0 release
@@ -21,6 +22,10 @@ java {
 
 repositories {
   mavenCentral()
+  mavenLocal()
+  maven {
+    url "https://bed-artifactory.bedford.progress.com:443/artifactory/ml-maven-snapshots/"
+  }
 }
 
 configurations {
@@ -42,7 +47,7 @@ dependencies {
     exclude module: "rocksdbjni"
   }
 
-  shadowDependencies ("com.marklogic:marklogic-client-api:6.6.1") {
+  shadowDependencies ("com.marklogic:marklogic-client-api:7.0.0") {
     // The Java Client uses Jackson 2.15.2; Scala 3.4.x does not yet support that and will throw the following error:
     // Scala module 2.14.2 requires Jackson Databind version >= 2.14.0 and < 2.15.0 - Found jackson-databind version 2.15.2
     // So the 4 Jackson modules are excluded to allow for Spark's to be used.
@@ -63,20 +68,37 @@ dependencies {
 
   shadowDependencies "org.jdom:jdom2:2.0.6.1"
 
-  testImplementation ('com.marklogic:ml-app-deployer:4.7.0') {
+  testImplementation ('com.marklogic:ml-app-deployer:4.8.0') {
     exclude group: "com.fasterxml.jackson.core"
     exclude group: "com.fasterxml.jackson.dataformat"
+
+    // Use the Java Client declared above.
+    exclude module: "marklogic-client-api"
   }
+
   testImplementation ('com.marklogic:marklogic-junit5:1.4.0') {
     exclude group: "com.fasterxml.jackson.core"
     exclude group: "com.fasterxml.jackson.dataformat"
+
+    // Use the Java Client declared above.
+    exclude module: "marklogic-client-api"
   }
 
   testImplementation "ch.qos.logback:logback-classic:1.3.14"
   testImplementation "org.slf4j:jcl-over-slf4j:1.7.36"
   testImplementation "org.skyscreamer:jsonassert:1.5.1"
 }
 
+// See https://jeremylong.github.io/DependencyCheck/dependency-check-gradle/configuration.html for more information.
+dependencyCheck {
+  // Need a JSON report to integrate with Sonar. And HTML is easier for humans to read.
+  formats = ["HTML", "JSON"]
+  // We don't include compileOnly since that includes Spark, and Spark and its dependencies are not actual dependencies
+  // of our connector.
+  scanConfigurations = ["shadowDependencies"]
+  suppressionFile = "config/dependency-check-suppressions.xml"
+}
+
 test {
   useJUnitPlatform()
   finalizedBy jacocoTestReport
@@ -95,6 +117,8 @@ sonar {
   properties {
     property "sonar.projectKey", "marklogic-spark"
     property "sonar.host.url", "http://localhost:9000"
+    // See https://github.com/dependency-check/dependency-check-sonar-plugin for more information.
+    property "sonar.dependencyCheck.jsonReportPath", "build/reports/dependency-check-report.json"
   }
 }
 

config/dependency-check-suppressions.xml

@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<suppressions xmlns="https://jeremylong.github.io/DependencyCheck/dependency-suppression.1.3.xsd">
+  <suppress>
+    <notes><![CDATA[
+   file name: jackson-databind-2.14.3.jar
+
+   See https://nvd.nist.gov/vuln/detail/CVE-2023-35116 and https://github.com/FasterXML/jackson-databind/issues/3972 .
+   The Jackson team heartily refutes that this is a vulnerability, and we agree.
+   ]]></notes>
+    <packageUrl regex="true">^pkg:maven/com\.fasterxml\.jackson\.core/jackson-databind@.*$</packageUrl>
+    <cve>CVE-2023-35116</cve>
+  </suppress>
+  <suppress>
+    <notes><![CDATA[
+   file name: commons-compress-1.24.0.jar
+   This is brought in by Jena 4.10. It's a medium, and we don't want to interfere with Jena dependencies.
+   ]]></notes>
+    <packageUrl regex="true">^pkg:maven/org\.apache\.commons/commons-compress@.*$</packageUrl>
+    <cve>CVE-2024-25710</cve>
+  </suppress>
+  <suppress>
+    <notes><![CDATA[
+   file name: commons-compress-1.24.0.jar
+   This is brought in by Jena 4.10. It's a medium, and we don't want to interfere with Jena dependencies.
+   ]]></notes>
+    <packageUrl regex="true">^pkg:maven/org\.apache\.commons/commons-compress@.*$</packageUrl>
+    <cve>CVE-2024-26308</cve>
+  </suppress>
+</suppressions>

docker-compose.yaml

@@ -28,9 +28,10 @@ services:
       SONAR_JDBC_USERNAME: sonar
       SONAR_JDBC_PASSWORD: sonar
     volumes:
-      - sonarqube_data:/opt/sonarqube/data
-      - sonarqube_extensions:/opt/sonarqube/extensions
-      - sonarqube_logs:/opt/sonarqube/logs
+      - ./docker/sonarqube/data:/opt/sonarqube/data
+      - ./docker/sonarqube/logs:/opt/sonarqube/logs
+      # Allows for Sonar plugins to be installed by including plugin jar files in this directory.
+      - ./docker/sonarqube/extensions:/opt/sonarqube/extensions
     ports:
       - "9000:9000"
 

docker/sonarqube/extensions/plugins/sonar-dependency-check-plugin-4.0.1.jar

@@ -32,15 +32,15 @@ connector and also to initialize Spark:
 
 ```
 import os
-os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars "/path/to/marklogic-spark-connector-2.2.0.jar" pyspark-shell'
+os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars "/path/to/marklogic-spark-connector-2.3.1.jar" pyspark-shell'
 
 from pyspark.sql import SparkSession
 spark = SparkSession.builder.master("local[*]").appName('My Notebook').getOrCreate()
 spark.sparkContext.setLogLevel("WARN")
 spark
 ```
 
-The path of `/path/to/marklogic-spark-connector-2.2.0.jar` should be changed to match the location of the connector 
+The path of `/path/to/marklogic-spark-connector-2.3.1.jar` should be changed to match the location of the connector 
 jar on your filesystem. You are free to customize the `spark` variable in any manner you see fit as well. 
 
 Now that you have an initialized Spark session, you can run any of the examples found in the 

docs/getting-started/jupyter.md

@@ -29,7 +29,7 @@ shell by pressing `ctrl-D`.
 
 Run PySpark from the directory that you downloaded the connector to per the [setup instructions](setup.md):
 
-    pyspark --jars marklogic-spark-connector-2.2.0.jar
+    pyspark --jars marklogic-spark-connector-2.3.1.jar
 
 The `--jars` command line option is PySpark's method for utilizing Spark connectors. Each Spark environment should have
 a similar mechanism for including third party connectors; please see the documentation for your particular Spark