Sample Text Data Preprocessing Implementation In SparkNLP

In this study, I will create all annotators one by one and subsequently collect them into one pipeline for completing the pre-processing. Here is which annotators I will create:->DocumentAssembler->Tokenizer->SentenceDetector->Normalizer->StopWordsCleaner->TokenAssembler->Stemmer->Lemmatizer

Importing SparkNLP, necessary libraries, reading the data from local and converting into spark dataFrame like following:

Annotators and transformers are come from base and annotator libraries. I won’t give details what is annotators and transformers throughout article.

We have two columns which names are category and message. Text column consist messages and category column consist type of messages, spam or not(ham).

1- Document Assembler

DocumentAssembler is the begining part of the any SparkNLP project. It creates the first annotation of type Document which may be used by annotators down the road. documentAseembler() comes from SparkNLP’s base class. We can use it like following:

Parameters:setInputCol() -> The name of the column that will be converted. We can specify only one column here. The significant point here It can read either a String column or an Array[String]. setOutputCol() -> (optional) The name of the column in Document type that is generated. We can specify only one column here. Default is ‘document’.setCleanupMode() -> (optional) Cleaning up options.

I choiced shrink as clean up mode, it removes new lines and tabs, plus merging multiple spaces and blank lines to a single space.Now, we will transform the df with documentAssembler by using transform() function and subsequently print the schema as following:

The annotators and transformers all come with universal metadata in SparkNLP. We can access all these parameters shown above with {column name}.{parameter name}.

In order to view text line by line and beginning and ending of the lines:

We can print out the first item’s result:

2- Tokenizer

Tokenizer() uses for identifying the tokens in SparkNLP. Here is to form tokenizer:

Also, tokenizer function provides so many parameters to make our task conveniently. For example;

setExceptions(StringArray): If you have a list of some composite words that are you don’t want to split, this is useful for you. contextChars(StringArray): If you don’t want to split some characters such as parentheses, question marks etc. it is useful. It needs string array. setTargetPattern() : If you want to identify a candidate for tokenization within basic regex rules, it is useful for you. Defaults to \S+ which means anything except a space.

These are general parameters, there are many parameters to use case by case.

3- Sentence Detector

SentenceDetector() finds a sentence bounds in raw text. Here is how I formed it:

An useful parameter here:setCustomBounds(string): If you want to separate sentences by custom characters, it is for you.

4- Normalizer

normalizer() cleans dirty characters after regex pattern and removes words based on given dictionary. Implementation like following:

Now, let’s explain the parameters.

.setLowercase(True) : It’s lowercasing the tokens, as a default false. .setCleanupPatterns(["[^\w\d\s"]) : It needs regular expressions list for normalization, as a default [^A-Za-z]. If you determine like I did above, it will remove punctuations, keep alphanumeric chars.

5- Stopwords Cleaner

StopWordsCleaner() uses for dropping the stopwords from text. Here is the implementation:

Explaining of parameters:.setCaseSensitive(True) : Whether to do a case sensitive comparison over the stop words. .setStopWords() : If you have the words to be filtered out. It needs Array[String].

6- Token Assembler

TokenAssembler() uses for assembling back again the cleaned tokens. Implementation like following:

7- Stemmer

The goal of stemming is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form. Here is the implementation:

8- Lemmatizer

Both in stemming and in lemmatization, we are reducing given word to it’s root. However, there are some differences between that. In stemming, the algorithms don’t actually know the meaning of the word in the language it belongs to. In lemmatization, the algorithms have this knowledge. These algorithms refer a dictionary to understand the meaning of the word before reducing. For example, lemmatization algorithms know that the word “went” is derived from the word “go” and hence, the lemma will be “go”. But a stemming algorithm wouldn’t be able to do the same. There could be over-stemming or under-stemming, and word “went” could be reduced to either “wen” or keep it as “went”. So, here is the implementation of Lemmatizer() :

We will first pull the lemmatization dictionary from link. Then, implement the lemmatization.

We selected the dictionary with the .setDictionary() parameter.

Fortunately, all the SparkNLP annotators and transformers can be used within Spark ML Pipelines. We just created annotators and transformers that we need. Now, we will put them into a pipeline. After that, we will fit the pipeline and transform it with our dataset. Let’s start!

Importing spark ml pipeline: from pyspark.ml import Pipeline

Putting annotators and transformers into the pipeline:

Now, in order to using the pipeline with different dataframes we will create an empty dataframe to fitting the pipeline.

Nice, we just builded model. It’s time to apply it to our dataset. For do this, we will use transform() function like following;

result= model.transform(df)

Examining The Results

Well, we will examine the results that what we did. But first, we will import functions library from pyspark sql in order to making some process. from pyspark.sql import functions as F

Let’s begin the examination with tokens and normalized tokens.

Tokens that seems in the first column has normalized in the second column. Let’s check the cleared data from stopwords like following;

As you see above, dropped some stopwords by stopwords cleaner.

Token assembler contains normalized tokens. Now, let’s compare the sentence detector result and token assembler result.

Also, we can check the sentence numbers of the words like following;

Seems nice! Now we will compare the tokens, stems and lemmas. Also, in this part we will see how to covert spark dataframe into pandas data frame easily.

You can compare stems and lemmas nicely. For “available” word, stem chanced it as “avail” because stemmmer algorithm doesn’t know what words mean.

Well, we’ve seen some basic preprocessing steps throughout this article. I suggest you visit JohnSnowLabs that is official developer of SparkNLP in order to access more info and detail. Also, there is great introduction as colab notebook .

Thanks for your read and support!