NLP Cheat Sheet — Tokenization, Embeddings, Attention, BERT, Transformers | Dataplexa

NLP Cheat Sheet

tokenize · embeddings · attention · BERT · transformers · text preprocessing

Sheet 1 of 4 Specialized ML Intermediate Printable

NLP Pipeline

Overview

A typical NLP pipeline transforms raw text into structured predictions through sequential processing stages.

Raw Text — input string, potentially noisy or multilingual
Preprocessing — lowercase, strip HTML, normalize unicode
Tokenization — split text into tokens (words, subwords)
Vectorization — convert tokens to numeric representations
Model — RNN, CNN, Transformer, or pretrained LM
Task Head — classification, NER, QA, generation layer
Output — labels, spans, generated text, or scores

Key insight: Modern pipelines (BERT/GPT) collapse steps 3–6 — the pretrained model handles tokenization, vectorization, and encoding jointly.

Tokenization

Text Splitting

Method	Unit	Used By
Word	whitespace split	Classic NLP
BPE	subword merges	GPT, RoBERTa
WordPiece	subword likelihood	BERT
SentencePiece	language-agnostic	T5, mBERT
Character	single char	CharCNN

Python — HuggingFace Tokenizer

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
enc = tok("Hello world!", return_tensors="pt")
# enc.input_ids  → token IDs
# enc.attention_mask → 1 for real, 0 for pad

[CLS] prepended for classification; [SEP] separates segments; [PAD] fills to max length.

Text Preprocessing

Classic NLP

Common Steps (NLTK / spaCy)

import nltk, re
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer

text = "Running faster than expected!"

# 1. Lowercase
text = text.lower()

# 2. Remove punctuation / special chars
text = re.sub(r'[^a-z\s]', '', text)

# 3. Tokenize
tokens = nltk.word_tokenize(text)

# 4. Remove stopwords
stops = set(stopwords.words('english'))
tokens = [t for t in tokens if t not in stops]

# 5. Lemmatize
lem = WordNetLemmatizer()
tokens = [lem.lemmatize(t) for t in tokens]
# → ['run', 'fast', 'expect']

Technique	Input → Output
Stemming	running → run (suffix strip)
Lemmatization	better → good (dict-based)
Stop-word removal	drop "the", "is", "a"
N-grams	"new york" → bigram feature

Word Embeddings

Vectorization

Dense vector representations that capture semantic similarity. Similar words cluster in embedding space.

Word2Vec — Skip-Gram Objective

maximize Σ log P(w_context | w_center)

Predict surrounding words given a center word. Learns by co-occurrence.

Cosine Similarity

sim(A, B) = (A · B) / (‖A‖ · ‖B‖)

Range [−1, 1]. Values near 1 = semantically similar words.

Model	Dim	Key Feature
Word2Vec	100–300	Static, fast training
GloVe	50–300	Global co-occurrence matrix
FastText	300	Subword — handles OOV
BERT Emb.	768	Contextual, dynamic

Static vs Contextual: Word2Vec gives "bank" one vector. BERT gives different vectors for "river bank" vs "bank account".

Attention Mechanism

Core Concept

Attention allows a model to focus on relevant parts of the input when producing each output token. It computes a weighted sum of values based on query–key similarity.

Scaled Dot-Product Attention

Attention(Q, K, V) = softmax( QKᵀ / √d_k ) · V

Q = Query matrix, K = Key matrix, V = Value matrix, d_k = key dimension. Scaling by √d_k prevents vanishing gradients with large dimensions.

Multi-Head Attention

MultiHead(Q,K,V) = Concat(head₁,...,head_h) W^O

Each head attends to different positions. Enables learning diverse relationships simultaneously. Typical: 8 or 12 heads.

Attention Type	Q source	K/V source	Used In
Self-Attention	same seq	same seq	Encoder/Decoder
Cross-Attention	decoder	encoder out	Seq2Seq, T5
Causal (Masked)	same seq	past tokens only	GPT, decoder
Sparse Attention	local window	key positions	Longformer

PyTorch — Scaled Dot-Product Attention

import torch, torch.nn.functional as F
import math

def attention(Q, K, V, mask=None):
    d_k = Q.size(-1)
    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
    weights = F.softmax(scores, dim=-1)
    return torch.matmul(weights, V), weights
# Built-in: F.scaled_dot_product_attention(Q, K, V)

Attention is O(n²) in sequence length — the bottleneck for long documents. Flash Attention and sparse variants address this.

Transformer Architecture

Architecture

Transformers replaced RNNs by processing all tokens in parallel using self-attention + feedforward layers.

Component	Purpose
Token Embedding	Maps token IDs → dense vectors
Positional Encoding	Injects sequence order information
Multi-Head Attention	Models token relationships
Layer Norm	Stabilises training, pre or post
Feed-Forward	2-layer MLP per position
Residual Connection	x + Sublayer(x) — prevents degradation

Sinusoidal Positional Encoding

PE(pos,2i) = sin(pos / 10000^(2i/d_model))

Alternating sin/cos at different frequencies. Allows model to attend to relative positions. BERT uses learned positional embeddings instead.

Encoder-only (BERT): bidirectional context, best for classification/NER. Decoder-only (GPT): causal, best for generation. Enc-Dec (T5, BART): best for translation/summarization.

BERT

Pretrained LM

Bidirectional Encoder Representations from Transformers. Pretrained on masked language modeling and next sentence prediction.

Masked Language Model (MLM)

mask 15% tokens → predict originals

80% [MASK], 10% random, 10% unchanged. Forces bidirectional context.

Next Sentence Prediction (NSP)

P(B follows A) → binary classification

Trains [CLS] token for sentence-pair tasks. Dropped in RoBERTa.

Fine-tuning BERT for Classification

from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained(
    "bert-base-uncased", num_labels=2
)
outputs = model(**enc)
loss = outputs.loss   # cross-entropy with labels
logits = outputs.logits  # [batch, num_labels]

Variant	Params	Difference
BERT-base	110M	12 layers, 768 hidden
BERT-large	340M	24 layers, 1024 hidden
RoBERTa	125M	No NSP, more data
DistilBERT	66M	60% faster, 97% accuracy
ALBERT	12M	Parameter sharing

Seq2Seq & Language Models

Generation

Model	Type	Trained On	Best For
GPT-2/3/4	Decoder	Next token pred	Text generation
T5	Enc-Dec	Text-to-text	Translation, QA, Summ.
BART	Enc-Dec	Denoising	Summarization
mT5	Enc-Dec	101 languages	Multilingual

Decoding Strategies

# Greedy — fastest, repetitive
model.generate(ids, max_length=50)

# Beam search — quality/diversity balance
model.generate(ids, num_beams=5, early_stopping=True)

# Sampling with temperature
model.generate(ids, do_sample=True,
    temperature=0.8, top_p=0.92)

Temperature: <1 = more focused/deterministic. >1 = more random/creative. top-p (nucleus): sample from smallest set of tokens with cumulative prob ≥ p.

NER & Sequence Labeling

Structured Output

Tag	Meaning	Example
`B-PER`	Begin Person	"Barack"
`I-PER`	Inside Person	"Obama"
`B-ORG`	Begin Org	"Google"
`B-LOC`	Begin Location	"Paris"
`O`	Outside entity	"the", "is"

spaCy NER

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple was founded by Steve Jobs in Cupertino.")
for ent in doc.ents:
    print(ent.text, ent.label_)
# Apple ORG | Steve Jobs PERSON | Cupertino GPE

BIO tagging scheme is standard. IOB2 (B always starts entity) is most common. BERT-based NER adds a token classification head on top of embeddings.

Text Classification

Tasks

TF-IDF + Logistic Regression

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

pipe = Pipeline([
    ('tfidf', TfidfVectorizer(max_features=50000)),
    ('clf',  LogisticRegression())
])
pipe.fit(X_train, y_train)

TF-IDF Formula

TF-IDF(t,d) = TF(t,d) × log(N / df(t))

TF = term frequency in doc. df = docs containing term. N = total docs. Downweights common terms.

Task	Output
Sentiment	pos/neg/neutral
Topic	category label
Intent	action class
Spam	binary

NLP Evaluation Metrics

Metrics

Metric	Task	Formula / Notes
Accuracy	Classification	correct / total
F1-Score	NER, Classification	2PR / (P+R)
BLEU	Translation	n-gram precision vs reference
ROUGE-L	Summarization	LCS recall vs reference
Perplexity	Language Model	exp(−(1/N) Σ log P(wᵢ))
EM / F1	QA (SQuAD)	exact match + partial token overlap
BERTScore	Generation	cosine sim of BERT embeddings

Perplexity — Lower is Better

PPL = exp(−(1/N) Σ_i=1^N log P(w_i | w_<i))

Measures how "surprised" a model is by test data. A random model over 10k vocab ≈ PPL 10,000. GPT-3 achieves ~20 on Penn Treebank.

NLP Libraries & Tools

Ecosystem

Library	Best For	Key Feature
`transformers`	BERT, GPT, T5	1000+ pretrained models
`spaCy`	NER, POS, parse	Production-speed pipelines
`NLTK`	Teaching, research	Corpora, tokenizers
`gensim`	Topic models	LDA, Word2Vec, Doc2Vec
`datasets`	Benchmark data	HuggingFace Hub datasets
`sentence-transformers`	Semantic search	Sentence embeddings
`LangChain`	LLM apps	Chains, RAG, agents

Semantic Similarity — sentence-transformers

from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer('all-MiniLM-L6-v2')
embs = model.encode(["cat on mat", "feline on rug"])
score = util.cos_sim(embs[0], embs[1])
# → tensor([[0.78]])  high similarity

Fine-Tuning & Transfer Learning

Practical Guide

Choose a pretrained base model (BERT, RoBERTa, T5) appropriate for task and language
Prepare dataset: tokenize with model's own tokenizer, keep max_length ≤ 512 for BERT
Add task head: classification layer on [CLS] token, or token-level for NER/QA
Freeze base layers (optional) for small datasets; unfreeze for large datasets
Train with low learning rate: 2e-5 to 5e-5 for transformers (much lower than training from scratch)
Use linear warmup + decay scheduler; batch size 16–32; 3–10 epochs
Evaluate on dev set; early stopping on validation loss

Catastrophic forgetting: Fine-tuning too aggressively overwrites pretrained knowledge. Use low LR and optionally PEFT methods (LoRA, adapters) to preserve base representations.

Full Fine-Tune with HuggingFace Trainer

from transformers import TrainingArguments, Trainer

args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    warmup_ratio=0.1,
    learning_rate=3e-5,
    weight_decay=0.01,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    fp16=True,          # mixed precision — 2× faster
)

trainer = Trainer(
    model=model, args=args,
    train_dataset=train_ds,
    eval_dataset=val_ds,
    compute_metrics=compute_metrics,
)
trainer.train()

Strategy	When	Trade-off
Full fine-tune	Large labelled data	Best perf, high cost
Frozen base	Very small data	Fast, may underfit
LoRA / QLoRA	Limited GPU memory	Near full-tune at 10× less VRAM
Prompt tuning	No labels	In-context learning only

NLP Mastery Checklist

Self-Assessment

Foundations

Explain tokenization strategies: word, BPE, WordPiece
Describe the NLP preprocessing pipeline end-to-end
Calculate TF-IDF for a term in a corpus
Compare static vs contextual embeddings
Implement Word2Vec training objective
Use cosine similarity for word/sentence comparison

Transformers & BERT

Derive scaled dot-product attention formula
Explain multi-head attention and its benefits
Distinguish encoder-only vs decoder-only vs enc-dec models
Describe BERT's MLM and NSP pretraining tasks
Fine-tune BERT for text classification with HuggingFace
Choose correct BERT variant for your use case

Applications & Evaluation

Apply BIO tagging for NER tasks
Implement semantic search with sentence-transformers
Select appropriate metric: BLEU, ROUGE, F1, perplexity
Apply LoRA or adapters for parameter-efficient fine-tuning
Control generation quality with temperature and top-p
Choose the right NLP library for production vs research

Next up in Specialized ML → Sheet 2 covers Computer Vision: YOLO, object detection, image segmentation, data augmentation, and OCR pipelines.

2 · Computer Vision Cheat Sheet →

← Back