AI Lesson 72 – BERT - Basics & Architecture | Dataplexa

AI Course

I. AI Foundations
1. Introduction to AI 2. History of AI 3. AI vs ML vs DL 4. Search Algorithms 5. Intelligent Agents 6. Informed & Uninformed Search 7. Constraint Satisfaction 8. Adversarial Search 9. Logic in AI 10. Planning in AI 11. Knowledge Representation 12. Probabilistic AI 13. Bayesian Networks 14. Markov Models 15. AI Ethics 16. AI Applications 17. Intro to PyTorch 18. Intro to TensorFlow 19. AI Workflows 20. Data for AI
II. Machine Learning
21. Introduction to ML 22. ML Workflow 23. Supervised Learning 24. Unsupervised Learning 25. Reinforcement Learning 26. Linear Regression 27. Logistic Regression 28. SVM 29. Decision Trees 30. Random Forest 31. Gradient Boosting 32. XGBoost 33. K-Means 34. Hierarchical Clustering 35. Dimensionality Reduction 36. Feature Engineering 37. Feature Selection 38. Model Evaluation 39. Overfitting vs Underfitting 40. Cross Validation
III. Deep Learning
41. Intro to Deep Learning 42. Neural Networks 43. Activation Functions 44. Backpropagation 45. Loss & Optimizers 46. Regularization 47. CNN Basics 48. CNN Architectures 49. RNN, LSTM, GRU 50. Seq2Seq 51. Transformers 52. Attention 53. Positional Encoding 54. Training DL 55. Hyperparameter Tuning 56. Transfer Learning 57. Autoencoders 58. GAN Basics 59. GAN Applications 60. Model Serving
IV. Natural Language Processing
61. Intro to NLP 62. Text Processing 63. Tokenization 64. Stopwords & Lemmatization 65. BoW & TF-IDF 66. Word Embeddings 67. Sentence Embeddings 68. NLP Classification 69. Sequence Modeling 70. RNN in NLP 71. Transformers in NLP 72. BERT Basics 73. BERT Fine-tuning 74. RoBERTa & DistilBERT 75. Sentiment Analysis 76. NER 77. Machine Translation 78. Text Generation 79. NLP Evaluation 80. NLP Use Cases
V. Computer Vision
81. Intro to CV 82. Image Processing 83. Filters & Edges 84. Convolutions 85. Object Detection 86. Object Tracking 87. Image Segmentation 88. Face Recognition 89. Feature Detection 90. OCR 91. Image Augmentation 92. Generative Vision 93. Vision Transformers 94. Multimodal Models 95. CV Use Cases
VI. LLM Engineering
96. Intro to LLMs 97. Tokenization 98. LLM Architecture 99. Pretraining 100. Fine-tuning 101. RLHF 102. Prompt Engineering 103. Embedding Models 104. Vector Databases 105. LLM Agents 106. Guardrails & Safety 107. AI in Production 108. End-to-End Project

Lesson 72: BERT Basics

In the previous lesson, we learned how transformers changed Natural Language Processing by using attention and parallel processing. In this lesson, we focus on one of the most important transformer-based models ever created — BERT.

BERT is designed to understand language deeply. Instead of predicting the next word like some models, BERT focuses on understanding the meaning of text as a whole.

Real-World Connection

Whenever Google understands your search query better, suggests the right result, or highlights the exact answer inside a long article, BERT is working behind the scenes.

BERT is widely used in search engines, question-answering systems, document classification, and enterprise NLP tools.

What Is BERT?

BERT stands for Bidirectional Encoder Representations from Transformers. It is an encoder-only transformer model that reads text in both directions — left to right and right to left — at the same time.

This bidirectional reading allows BERT to understand context more accurately than earlier models.

  • Uses transformer encoder architecture
  • Understands full sentence context
  • Pretrained on massive text data

Why Bidirectional Understanding Matters

Consider the sentence: “The bank was crowded.” Without context, the word “bank” is ambiguous. BERT reads surrounding words on both sides to understand whether it refers to a financial institution or a river bank.

This bidirectional context is the key reason BERT outperforms traditional NLP models.

How BERT Is Trained

BERT is pretrained using two main tasks:

  • Masked Language Modeling: Random words are hidden, and the model predicts them
  • Next Sentence Prediction: The model learns sentence relationships

These tasks teach BERT grammar, semantics, and sentence-level understanding.

Simple BERT Example

Below is a basic example of using a pretrained BERT model for text classification. 


from transformers import pipeline

classifier = pipeline("text-classification", model="bert-base-uncased")

text = "Dataplexa makes learning AI simple and practical"
result = classifier(text)

print(result)
[{'label': 'POSITIVE', 'score': 0.98}]

Understanding the Code

The pipeline loads a pretrained BERT model. The input sentence is tokenized and passed through multiple transformer encoder layers.

Each word attends to every other word, allowing BERT to understand meaning in context. The output shows the predicted label and confidence score.

What BERT Is Best At

  • Text classification
  • Question answering
  • Named Entity Recognition
  • Semantic search

BERT vs GPT (High Level)

BERT focuses on understanding text, while GPT focuses on generating text. BERT is encoder-only, whereas GPT is decoder-only.

Because of this, BERT is usually chosen for tasks where understanding is more important than generation.

Limitations of BERT

  • Cannot generate long text
  • Computationally expensive
  • Limited input length

Practice Questions

Practice 1: How does BERT read text?



Practice 2: Which training task hides words during training?



Practice 3: BERT is based on which transformer component?



Quick Quiz

Quiz 1: Which model is mainly used for text understanding?





Quiz 2: What does BERT predict during masked language modeling?





Quiz 3: What type of transformer architecture does BERT use?





Coming up next: BERT Fine-Tuning — adapting pretrained models to real-world tasks.

← Previous Course Index Next →