Do lemmatization for each file in a folder using nltk in python.
Prerequisite:
installed the [NLTK python package] (http://nltk.org/install.html)
Downloaded the WordNet corpora by: python -m nltk.downloader all
To run:
python lemmatizationWithNLTK.py <input folder path> <output folder path>
For example:
python lemmatizationWithNLTK.py .\\inputData .\\outputData
Why lemmatization and not stemming? To quote my Master's thesis:
We lemmatize all the words to reduce the inflectional forms. English words usually have more than one form with the same semantic meanings, for example, car and cars. To reduce the forms to their base forms helps us in building the keyword graph and the community mining process later. Both stemming and lemmatization could achieve this goal. Many researches use stemming because it is easy to do. Stemming methods usually just chop off the end of words according to a set of brutal heuristics. Lemmatization, on the other hand, is more reasonable. It utilizes dictionaries and morphological information, aiming to remove only the inflectional endings rather than chopping a large part off from the words [50]. For example, the word large is stemmed to larg with the famous Porter Stemmer but it is kept intact with the WordNet Lemmatizer. Therefore, in this work we use the WordNet Lemmatizer provided with the Natual Language Toolkit (NLTK) 2 to get the lemma for each word. A lemma is usually the base form of a word, just like how the word would appear in a dictionary. We also store the original words along with their lemmas. At the post-processing stage discussed in Section 4.5.7, we transform the lemmas back to their most frequent original form so that they make more sense to the users [72].