252 release candidate

[maziyarpanahi]

2.5.2

---

New Features

• Introducing a new LanguageDetectorDL state-of-the-art annotator to detect and identify languages in documents and sentences
• Add a new param entityValue to TextMatcher to add custom value inside metadata. Useful in post-processing when there are multiple TextMatcher annotators with multiple dictionaries Add custom metadata to annotators #920

---

Bugfixes

• Add missing TensorFlow graphs to train ContextSpellChecker annotator Pre-defined graphs for ContextSpellChecker are missing #912
• Fix misspelled param in classThreshold param in ContextSpellChecker annotator Misspelled param name in ContextSpellChecker #911
• Fix a bug where setGraphFolder in NerDLApproach annotator couldn't find a graph on Databricks (DBFS) NerDLApproach does not accept HDFS path for graph folder #739
• Fix a bug in NerDLApproach when includeConfidence was set to true java.lang.NumberFormatException BigDecimal in NerDLApproach regarding includeConfidence #917
• Fix a bug in BertEmbeddings BertEmbeddings NullPointerException in Python #906 NullPointerException using BertEmbeddings #918

---

Enhancements

• Improve TF backend in ContextSpellChecker annotator

---

Pipelines and Models

We have added 4 new LanguageDetectorDL models and pipelines to detect and identify up to 20 languages:

• The model with 7 languages: Czech, German, English, Spanish, French, Italy, and Slovak
• The model with 20 languages: Bulgarian, Czech, German, Greek, English, Spanish, Finnish, French, Croatian, Hungarian, Italy, Norwegian, Polish, Portuguese, Romanian, Russian, Slovak, Swedish, Turkish, and Ukrainian

Model	Name	Build	Lang	Offline
LanguageDetectorDL	ld_wiki_7	2.5.2	xx	Download
LanguageDetectorDL	ld_wiki_20	2.5.2	xx	Download

Pipeline	Name	Build	Lang	Offline
LanguageDetectorDL	detect_language_7	2.5.2	xx	Download
LanguageDetectorDL	detect_language_20	2.5.2	xx	Download

---

Documentation

• Update documentation for release of Spark NLP 2.5.x
• Update the entire spark-nlp-workshop notebooks for Spark NLP 2.5.x
• Update the entire spark-nlp-models repository with new pre-trained models and pipelines

---

Installation

Python

#PyPI

pip install spark-nlp==2.5.2

#Conda

conda install -c johnsnowlabs spark-nlp==2.5.2

Spark

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.11:2.5.2

PySpark

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.11:2.5.2

Maven

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.11</artifactId>
    <version>2.5.2</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/spark-nlp-assembly-2.5.2.jar

• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/spark-nlp-gpu-assembly-2.5.2.jar