John Snow Labs Spark-NLP 4.2.4: Introducing support for GCP storage for pre-trained models, update to TensorFlow 2.7.4 with CVEs fixes, improvements, and bug fixes

[maziyarpanahi]

---

📢 Overview

Spark NLP 4.2.4 🚀 comes with new support for GCP storage to automatically download and load models & pipelines via setting the cache_pretrained path, update to TensorFlow 2.7.4 with security patch fixes, lots of improvements in our documentation, improvements, and bug fixes.

Do not forget to visit Models Hub with over 11400+ free and open-source models & pipelines. As always, we would like to thank our community for their feedback, questions, and feature requests. 🎉

---

⭐ New Features & improvements

• Introducing support for GCP storage to automatically download and load pre-trained models/pipelines from cache_pretrained directory
• Update to TensorFlow 2.7.4 with bug and CVEs fixes. Details about bugs and CVEs fixes: 417e2a1
• Improve error handling while importing external TensorFlow models into Spark NLP
• Improve error messages when importing external models from remote storages like DBFS, S3, and HDFS
• Update documentation on how to use testDataset param in NerDLApproach, ClassifierDLApproach, MultiClassifierDLApproach, and SentimentDLApproach
• Update installation instructions for the Apple M1 chip
• Add support for future decoder-encoder models with 2 separated models

---

🐛 Bug Fixes

• Add missing setPreservePosition in NerConverter
• Add missing inputAnnotatorTypes to BigTextMatcher, ViveknSentimentModel, and NerConverter annotators
• Fix all wrong example codes provided for LemmatizerModel in Models Hub
• Fix the t5_grammar_error_corrector model to be compatible with Spark NLP 4.0+
• Fix provided notebook to import Longformer models from Hugging Face into Spark NLP

📓 New Notebooks

Spark NLP	Notebooks	Colab
Spark NLP Conf	Dowbload and Load Model from GCP Storage
LongformerEmbeddings	HuggingFace in Spark NLP - Longformer

• You can visit Import Transformers in Spark NLP
• You can visit Spark NLP Workshop for 100+ examples

---

📖 Documentation

• TF Hub & HuggingFace to Spark NLP
• Models Hub with new models
• Spark NLP documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs
• Spark NLP Workshop notebooks
• Spark NLP publications
• Spark NLP in Action
• Spark NLP training certification notebooks for Google Colab and Databricks
• Spark NLP Display for visualization of different types of annotations
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!

---

Installation

Python

#PyPI

pip install spark-nlp==4.2.4

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.4

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.4

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.4

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.4

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.2.4</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.2.4</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.2.4</version>
</dependency>

spark-nlp-aarch64:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-aarch64 -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.2.4</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-4.2.4.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-4.2.4.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-m1-assembly-4.2.4.jar

• AArch64 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-4.2.4.jar

What's Changed

• release note v4.2.2 by @Cabir40 in release note v4.2.2 #13091
• added languages by @ahmedlone127 in added languages #13097
• [skip ci] Create PR 4.2.2-healthcare-docs-8fde8ce2327dce2fb89db1742eec8ca121eee0de-3 by @jsl-builder in [skip ci] Create PR 4.2.2-healthcare-docs-8fde8ce2327dce2fb89db1742eec8ca121eee0de-3 #13084
• FEATURE NMH-139: Add annotator to existing model [skip-test] by @KshitizGIT in FEATURE NMH-139: Add annotator to existing model [skip-test] #13096
• Add Visual NLP 4.2 to compatible versions in models.json by @pabla in Add Visual NLP 4.2 to compatible versions in models.json #13099
• Add new demos 25 by @agsfer in Add new demos 25 #13100
• Docs/alab 4.3.0 by @diatrambitas in Docs/alab 4.3.0 #13104
• Added content for installation in OpenShift by @suvrat-joshi in Added content for installation in OpenShift #13105
• Update subtabs by @agsfer in Update subtabs #13110
• Release Notes Updated by @Cabir40 in Release Notes Updated #13111
• Updated old hc snippets by @ArshaanNazir in Updated old hc snippets #13092
• Added content for healthcare nlp integration by @suvrat-joshi in Added content for healthcare nlp integration #13115
• Added some content for troubleshooting section by @suvrat-joshi in Added some content for troubleshooting section #13116
• Docs/alab 2479 add content for model testing page by @rpranab in Docs/alab 2479 add content for model testing page #13114
• Update oncology.md by @agsfer in Update oncology.md #13146
• SPARKNLP-656 & SPARKNLP-657: Updated Documentation by @DevinTDHa in SPARKNLP-656 & SPARKNLP-657: Updated Documentation #13108
• SPARKNLP-658 Update EngineError message by @maziyarpanahi in SPARKNLP-658 Update EngineError message #13109
• SPARKNLP-661: Add missing setPreservePosition in NerConverter by @DevinTDHa in SPARKNLP-661: Add missing setPreservePosition in NerConverter #13112
• fixed Wrong Example code provided for LemmatizerModel Wrong Example code provided for LemmatizerModel #13125 by @ahmedlone127 in fixed Wrong Example code provided for LemmatizerModel #13125 #13126
• SPARKNLP-620 Provide GCP Support for Cache Folder by @danilojsl in SPARKNLP-620 Provide GCP Support for Cache Folder #13141
• SPARKNLP-669 Adding missing inputAnnotatorTypes by @danilojsl in SPARKNLP-669 Adding missing inputAnnotatorTypes #13144
• SPARKNLP-665 Updating to TensorFlow 2.7.4 by @maziyarpanahi in SPARKNLP-665 Updating to TensorFlow 2.7.4 #13152
• SPARKNLP-671 incorporate the exception into the error message by @maziyarpanahi in SPARKNLP-671 incorporate the exception into the error message #13153
• Models hub by @maziyarpanahi in Models hub #13160
• Release/424 release candidate by @maziyarpanahi in Release/424 release candidate #13163

Full Changelog: 4.2.3...4.2.4

---

This discussion was created from the release John Snow Labs Spark-NLP 4.2.4: Introducing support for GCP storage for pre-trained models, update to TensorFlow 2.7.4 with CVEs fixes, improvements, and bug fixes.