Sentence Embeddings

Word embeddings helped machines understand individual words, but real meaning rarely exists at the word level.

Humans communicate using sentences, where meaning depends on word order, context, and relationships.

Sentence embeddings exist to capture this higher-level meaning.

Why Word Embeddings Are Not Enough

Consider these two sentences:

• “The bank approved the loan.”
• “The bank of the river flooded.”

The word bank appears in both, but the meaning is completely different.

A single word embedding cannot resolve this ambiguity.

Thinking Before Coding

Ask:

How can a model represent meaning that depends on multiple words together?

The answer is to embed the entire sentence as one vector.

What Is a Sentence Embedding?

A sentence embedding is a dense vector that represents the semantic meaning of a full sentence.

It encodes:

• Word meanings
• Word order
• Context and relationships

Two sentences with similar meaning will have vectors that are close in space.

Naive Approach: Averaging Word Embeddings

Before modern models, a simple way to create sentence embeddings was to average word embeddings.

Why Show This?

Because understanding weak approaches helps you appreciate modern solutions.

Simple Averaging Example

import numpy as np

word_vectors = {
    "ai": np.array([0.9, 0.1]),
    "changes": np.array([0.7, 0.3]),
    "everything": np.array([0.6, 0.4])
}

sentence = ["ai", "changes", "everything"]

sentence_embedding = np.mean([word_vectors[w] for w in sentence], axis=0)
print(sentence_embedding)
  

This works to some extent, but it ignores word order and deeper structure.

Limitations of Averaging

This approach fails when:

• Word order changes meaning
• Negation is present
• Long-range dependencies exist

Modern GenAI systems require more expressive models.

Modern Sentence Embeddings

Modern sentence embeddings are generated using transformer-based architectures.

Instead of embedding words independently, the model processes the entire sentence at once.

How Transformers Enable Sentence Meaning

Transformers use attention mechanisms to understand how words relate to each other.

This allows the model to:

• Capture context
• Handle ambiguity
• Represent full semantic meaning

Sentence Similarity in Practice

Sentence embeddings are most often used to compare meaning between sentences.

System Thinking

Before writing code, define the goal:

Do we want to search, cluster, or match meaning?

This determines how embeddings are used.

Similarity Comparison Example

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

s1 = np.array([0.8, 0.2])
s2 = np.array([0.75, 0.25])
s3 = np.array([0.1, 0.9])

print(cosine_similarity(s1, s2))
print(cosine_similarity(s1, s3))
  

Higher similarity indicates closer semantic meaning.

Where Sentence Embeddings Are Used

Sentence embeddings power many GenAI systems:

• Semantic search engines
• Chatbots and assistants
• Duplicate detection
• Question answering systems

They are foundational for RAG pipelines.

Sentence vs Word Embeddings

Key differences:

• Word embeddings capture isolated meaning
• Sentence embeddings capture contextual meaning
• Sentence embeddings are task-oriented

Most production systems prefer sentence-level embeddings.

Practice

Quick Quiz

Recap: Sentence embeddings represent full semantic meaning by encoding context, structure, and relationships.

Next up: We scale from sentences to entire documents — embedding long-form knowledge.