AI Lesson 64 – Stopwords, Stemming & Lemmatization | Dataplexa

AI Course

I. AI Foundations
1. Introduction to AI 2. History of AI 3. AI vs ML vs DL 4. Search Algorithms 5. Intelligent Agents 6. Informed & Uninformed Search 7. Constraint Satisfaction 8. Adversarial Search 9. Logic in AI 10. Planning in AI 11. Knowledge Representation 12. Probabilistic AI 13. Bayesian Networks 14. Markov Models 15. AI Ethics 16. AI Applications 17. Intro to PyTorch 18. Intro to TensorFlow 19. AI Workflows 20. Data for AI
II. Machine Learning
21. Introduction to ML 22. ML Workflow 23. Supervised Learning 24. Unsupervised Learning 25. Reinforcement Learning 26. Linear Regression 27. Logistic Regression 28. SVM 29. Decision Trees 30. Random Forest 31. Gradient Boosting 32. XGBoost 33. K-Means 34. Hierarchical Clustering 35. Dimensionality Reduction 36. Feature Engineering 37. Feature Selection 38. Model Evaluation 39. Overfitting vs Underfitting 40. Cross Validation
III. Deep Learning
41. Intro to Deep Learning 42. Neural Networks 43. Activation Functions 44. Backpropagation 45. Loss & Optimizers 46. Regularization 47. CNN Basics 48. CNN Architectures 49. RNN, LSTM, GRU 50. Seq2Seq 51. Transformers 52. Attention 53. Positional Encoding 54. Training DL 55. Hyperparameter Tuning 56. Transfer Learning 57. Autoencoders 58. GAN Basics 59. GAN Applications 60. Model Serving
IV. Natural Language Processing
61. Intro to NLP 62. Text Processing 63. Tokenization 64. Stopwords & Lemmatization 65. BoW & TF-IDF 66. Word Embeddings 67. Sentence Embeddings 68. NLP Classification 69. Sequence Modeling 70. RNN in NLP 71. Transformers in NLP 72. BERT Basics 73. BERT Fine-tuning 74. RoBERTa & DistilBERT 75. Sentiment Analysis 76. NER 77. Machine Translation 78. Text Generation 79. NLP Evaluation 80. NLP Use Cases
V. Computer Vision
81. Intro to CV 82. Image Processing 83. Filters & Edges 84. Convolutions 85. Object Detection 86. Object Tracking 87. Image Segmentation 88. Face Recognition 89. Feature Detection 90. OCR 91. Image Augmentation 92. Generative Vision 93. Vision Transformers 94. Multimodal Models 95. CV Use Cases
VI. LLM Engineering
96. Intro to LLMs 97. Tokenization 98. LLM Architecture 99. Pretraining 100. Fine-tuning 101. RLHF 102. Prompt Engineering 103. Embedding Models 104. Vector Databases 105. LLM Agents 106. Guardrails & Safety 107. AI in Production 108. End-to-End Project

Stopwords, Stemming, and Lemmatization

After tokenization, text still contains many words that may not contribute much meaning to analysis. Words like “is”, “the”, “and”, or “of” appear very frequently but usually do not help in understanding intent or sentiment. NLP uses techniques like stopword removal, stemming, and lemmatization to reduce noise and improve clarity.

This lesson explains why these techniques are used, how they differ, and how they are applied in real NLP systems.

Real-World Connection

When a search engine processes your query, it focuses on important words and often ignores filler words. For example, in the query “best phone to buy in 2026”, words like “to” and “in” are ignored, while “best”, “phone”, and “buy” matter. This filtering is done using stopwords and word normalization techniques.

What Are Stopwords?

Stopwords are common words that usually carry little meaning on their own. Removing them helps models focus on important terms.

  • Examples: is, the, and, a, an, of, to
  • Reduces vocabulary size
  • Improves processing speed

Stopword Removal Example


from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS

sentence = "This is a simple example of stopword removal"
words = sentence.lower().split()

filtered_words = [word for word in words if word not in ENGLISH_STOP_WORDS]
print(filtered_words)
['simple', 'example', 'stopword', 'removal']

What the Code Is Doing

The sentence is split into words, converted to lowercase, and common stopwords are removed. The remaining words carry more meaningful information.

What Is Stemming?

Stemming reduces words to their base form by removing suffixes. The resulting word may not always be a real dictionary word.

  • Fast and simple
  • May produce incorrect word forms
  • Useful for basic text analysis

Stemming Example


from nltk.stem import PorterStemmer

stemmer = PorterStemmer()
words = ["running", "runner", "ran", "runs"]

stems = [stemmer.stem(word) for word in words]
print(stems)
['run', 'runner', 'ran', 'run']

Understanding Stemming Output

Stemming cuts words mechanically. While fast, it may not always preserve the actual meaning of the word.

What Is Lemmatization?

Lemmatization reduces words to their meaningful base form, called a lemma. Unlike stemming, lemmatization considers grammar and context.

  • Produces real dictionary words
  • More accurate than stemming
  • Requires linguistic knowledge

Lemmatization Example


from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()
words = ["running", "better", "cars"]

lemmas = [lemmatizer.lemmatize(word) for word in words]
print(lemmas)
['running', 'better', 'car']

Stemming vs Lemmatization

  • Stemming is rule-based and fast
  • Lemmatization is grammar-aware and accurate
  • Stemming may produce non-words
  • Lemmatization produces valid words

When to Use Each Technique

The choice depends on the application:

  • Use stopwords for noise reduction
  • Use stemming for speed-focused tasks
  • Use lemmatization for semantic understanding

Practice Questions

Practice 1: What do we call common words removed from text?



Practice 2: Which technique removes suffixes mechanically?



Practice 3: Which technique returns dictionary words?



Quick Quiz

Quiz 1: Why are stopwords removed?





Quiz 2: Which method may produce non-dictionary words?





Quiz 3: Which technique considers grammar and meaning?





Coming up next: Bag of Words and TF-IDF — converting text into numerical features.

← Previous Course Index Next →