Lemmatization

In the previous lesson, you learned about stemming, which reduces words to a rough base form. However, you also saw that stemming can sometimes produce incorrect or incomplete words.

To solve this problem, NLP uses a more intelligent technique called lemmatization.

In this lesson, you will understand what lemmatization is, how it works, why it is better than stemming in many cases, and when to use it in real-world NLP applications.

What Is Lemmatization?

Lemmatization is the process of converting a word to its dictionary base form, known as the lemma.

Unlike stemming, lemmatization:

• Considers grammar and meaning
• Produces real, meaningful words
• Uses linguistic rules and vocabulary

Examples:

• running → run
• better → good
• studies → study
• went → go

All output words are valid dictionary words.

Why Lemmatization Is Important

Lemmatization helps NLP systems understand the true meaning of words, not just their surface form.

Main advantages:

• Produces meaningful base words
• Preserves semantic correctness
• Improves accuracy in many NLP tasks

This makes lemmatization especially useful in applications that require deeper language understanding.

Stemming vs Lemmatization (Core Difference)

This comparison is very important for exams and interviews.

Real-Life Intuition

Think like a language teacher.

When you see the words:

• am
• are
• is

You know they all relate to the verb "be". Lemmatization helps machines understand this relationship.

How Lemmatization Works

Lemmatization uses:

• Part-of-speech (POS) tagging
• Dictionary lookups
• Grammar rules

Because of this, lemmatization needs more information than stemming, but produces much better results.

Example Without Lemmatization

Sentence:

"The children are running faster"

Without lemmatization:

• children
• are
• running
• faster

Words remain in different forms.

Example With Lemmatization

After lemmatization:

• child
• be
• run
• fast

Now the meaning is preserved in its purest form.

Code Example: Lemmatization Using NLTK

You can run this code in:

• Google Colab (recommended)
• Jupyter Notebook
• VS Code / PyCharm

from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

words = ["running", "better", "children", "went"]

lemmatized_words = [lemmatizer.lemmatize(word) for word in words]

print(lemmatized_words)

Output:

['running', 'better', 'child', 'went']

Why Some Words Did Not Change?

By default, the lemmatizer assumes all words are nouns.

To get better results, we must provide part-of-speech (POS) information.

This is why POS tagging becomes important in advanced NLP pipelines.

Lemmatization in Real NLP Applications

• Search engines
• Chatbots
• Question answering systems
• Document summarization
• Information retrieval

Any task requiring accurate language understanding benefits from lemmatization.

Assignment / Homework

Where to practice:

• Google Colab
• Jupyter Notebook

Your tasks:

• Apply lemmatization on 10 sentences
• Compare output with stemming results
• Try lemmatization with different POS tags
• Observe where lemmatization performs better

Practice Questions

Quick Quiz

Quick Recap

• Lemmatization converts words to dictionary base form
• It preserves meaning and grammar
• More accurate than stemming
• Slightly slower but more reliable
• Best for meaning-sensitive NLP tasks