"TypeError: 'JavaPackage' object is not callable" when using PySpark 3.3.0 and sparknlp 4.0.2

[Colinnnnnm]

I am working with PySpark 3.3.0 and using latest sparknlp 4.0.2

import findspark
findspark.init()
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
from tqdm import tqdm
tqdm.pandas()
import pyspark
print(pyspark.__version__) # printed 3.3.0
spark = SparkSession.builder.getOrCreate()
from sparknlp.base import *
from sparknlp.annotator import *
document_assembler= DocumentAssembler().setInputCol("a").setOutputCol("b")

When calling document_assembler or any other module under sparknlp, I get the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 document_assembler= DocumentAssembler().setInputCol("a").setOutputCol("b")

File ~/C/SP/spark-3.3.0-bin-hadoop3/python/pyspark/__init__.py:135, in keyword_only.<locals>.wrapper(self, *args, **kwargs)
    133     raise TypeError("Method %s forces keyword arguments." % func.__name__)
    134 self._input_kwargs = kwargs
--> 135 return func(self, **kwargs)

File ~/anaconda3/envs/Text/lib/python3.9/site-packages/sparknlp/base/document_assembler.py:92, in DocumentAssembler.__init__(self)
     90 @keyword_only
     91 def __init__(self):
---> 92     super(DocumentAssembler, self).__init__(classname="com.johnsnowlabs.nlp.DocumentAssembler")
     93     self._setDefault(outputCol="document", cleanupMode='disabled')

File ~/C/SP/spark-3.3.0-bin-hadoop3/python/pyspark/__init__.py:135, in keyword_only.<locals>.wrapper(self, *args, **kwargs)
    133     raise TypeError("Method %s forces keyword arguments." % func.__name__)
    134 self._input_kwargs = kwargs
--> 135 return func(self, **kwargs)

File ~/anaconda3/envs/Text/lib/python3.9/site-packages/sparknlp/internal/annotator_transformer.py:33, in AnnotatorTransformer.__init__(self, classname)
     31 self.setParams(**kwargs)
     32 self.__class__._java_class_name = classname
---> 33 self._java_obj = self._new_java_obj(classname, self.uid)

File ~/C/SP/spark-3.3.0-bin-hadoop3/python/pyspark/ml/wrapper.py:86, in JavaWrapper._new_java_obj(java_class, *args)
     84     java_obj = getattr(java_obj, name)
     85 java_args = [_py2java(sc, arg) for arg in args]
---> 86 return java_obj(*java_args)

TypeError: 'JavaPackage' object is not callable

Here is my OS setting:

PRETTY_NAME="Ubuntu 22.04 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Java version:

openjdk version "11.0.16" 2022-07-19
OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu122.04)
OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu122.04, mixed mode, sharing)

Is there anyone face similar situation before and have an idea to solve it? Thanks in advanced.

[maziyarpanahi]

You forgot to include Spark NLP Maven package in your SparkSession:

• Either use the start function

spark = sparknlp.start()

And we have parameters to be set if needed: https://github.com/JohnSnowLabs/spark-nlp#packages-cheatsheet

• Or construct it manually (like you do) with either Maven coordinates or a JAR:

spark = SparkSession.builder \
    .appName("Spark NLP")\
    .master("local[*]")\
    .config("spark.driver.memory","16G")\
    .config("spark.driver.maxResultSize", "0") \
    .config("spark.kryoserializer.buffer.max", "2000M")\
    .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:4.0.2")\
    .getOrCreate()

https://github.com/JohnSnowLabs/spark-nlp#packages-cheatsheet

Extra examples: https://github.com/JohnSnowLabs/spark-nlp-workshop