Word Embeddings (Word2Vec and GloVe)

In the previous lesson, we converted text into numbers using Bag of Words and TF-IDF. While those techniques are useful, they treat words as independent units and ignore meaning and context. Word embeddings solve this limitation by representing words based on how they are used in language.

Word embeddings allow machines to understand relationships between words such as similarity, analogy, and semantic meaning. This lesson introduces word embeddings and explains two widely used techniques: Word2Vec and GloVe.

Real-World Connection

When you search for “king” on a smart search engine, it may also suggest “queen”, “prince”, or “royalty”. This happens because the system understands that these words are related in meaning. Word embeddings make this possible by placing similar words close together in a numerical space.

What Are Word Embeddings?

Word embeddings are dense numerical vectors that represent words in a continuous vector space. Words with similar meanings have similar vectors. Unlike Bag of Words, embeddings capture semantic relationships.

• Each word is represented as a vector of numbers
• Similar words have similar vectors
• Meaning is learned from context

Why Word Embeddings Are Powerful

Embeddings capture relationships that count-based models cannot. For example:

• king − man + woman ≈ queen
• Paris − France + Italy ≈ Rome

This ability makes embeddings extremely valuable for NLP tasks such as search, translation, and recommendation systems.

Word2Vec Overview

Word2Vec is a neural network–based technique that learns word embeddings from large text corpora. It is based on the idea that words appearing in similar contexts have similar meanings.

There are two main Word2Vec architectures:

• CBOW (Continuous Bag of Words): Predicts a word using surrounding context
• Skip-gram: Predicts surrounding words given a target word

Word2Vec Example

from gensim.models import Word2Vec

sentences = [
    ["ai", "is", "changing", "technology"],
    ["technology", "is", "driven", "by", "ai"],
    ["ai", "and", "data", "drive", "innovation"]
]

model = Word2Vec(sentences, vector_size=50, window=3, min_count=1)

vector = model.wv["ai"]
print(vector[:5])
  

Understanding the Output

The output is a numerical vector representing the word “ai”. Each value captures a different semantic dimension learned from surrounding words. These vectors are later used by machine learning models.

Finding Similar Words Using Word2Vec

similar_words = model.wv.most_similar("ai")
print(similar_words)
  

What GloVe Is

GloVe (Global Vectors) is another embedding technique that learns word vectors using global word co-occurrence statistics. Unlike Word2Vec, which focuses on local context, GloVe combines both local and global information.

• Uses word co-occurrence matrix
• Captures global statistics
• Often performs well on similarity tasks

Word2Vec vs GloVe

• Word2Vec learns from local context windows
• GloVe learns from global co-occurrence counts
• Both produce dense semantic vectors
• Both are widely used in NLP pipelines

Where Word Embeddings Are Used

• Search engines
• Recommendation systems
• Sentiment analysis
• Machine translation
• Chatbots and virtual assistants

Practice Questions

Quick Quiz

Coming up next: Sentence Embeddings — representing entire sentences instead of individual words.