Transformers
attention Β· positional encoding Β· encoder Β· decoder Β· BERT Β· GPT Β· T5 Β· fine-tuning
Sheet 4 of 4
Deep Learning
Advanced
Printable
Why Transformers?
Transformers replace recurrence with self-attention β processing all tokens in parallel and directly modelling relationships between any two positions regardless of distance.
| Problem with RNNs | Transformer Solution |
|---|---|
| Sequential β can't parallelise | All tokens processed in parallel |
| Vanishing gradient over long seqs | Direct attention β any two tokens connect |
| Fixed-size context vector bottleneck | Attention over all encoder states |
| Hard to scale to very long seqs | Scales with compute (O(nΒ²) attention) |
"Attention is All You Need" β Vaswani et al., 2017. The original Transformer paper introduced the architecture that powers BERT, GPT, T5, and every modern LLM.
Self-Attention
Each token attends to every other token in the sequence simultaneously. Three learned projections β Query, Key, Value β control what each token looks for and what it shares.
Scaled Dot-Product Attention
Attention(Q, K, V) = softmax(QKα΅ / βdβ) Β· V
Q = query Β· K = key Β· V = value Β· dβ = key dimension Β· βdβ prevents softmax saturation
| Matrix | Shape | Role |
|---|---|---|
| Q (Query) | (seq, dβ) | "What am I looking for?" |
| K (Key) | (seq, dβ) | "What do I have to offer?" |
| V (Value) | (seq, dα΅₯) | "What do I actually send?" |
| Output | (seq, dα΅₯) | Weighted sum of Values |
Complexity: Self-attention is O(nΒ²Β·d) β quadratic in sequence length. For very long sequences (n > 2048), efficient variants (Longformer, Flash Attention) are used.
Multi-Head Attention
Run h attention heads in parallel, each with its own Q, K, V projections β allowing the model to attend to different aspects of the sequence simultaneously (syntax, semantics, coreferenceβ¦).
Multi-Head Attention
MultiHead(Q,K,V) = Concat(headβ,β¦,headβ)Β·Wα΄Ό
headα΅’ = Attention(QΒ·Wα΅’α΄Ό, KΒ·Wα΅’α΄·, VΒ·Wα΅’α΅)
Each head uses dβ = dmodel/h Β· Wα΄Ό = output projection
| Model | d_model | Heads (h) | dβ per head |
|---|---|---|---|
| BERT-Base | 768 | 12 | 64 |
| BERT-Large | 1024 | 16 | 64 |
| GPT-2 Small | 768 | 12 | 64 |
| GPT-3 | 12288 | 96 | 128 |
| ViT-B/16 | 768 | 12 | 64 |
Positional Encoding
Transformers have no inherent order β attention is permutation invariant. Positional encoding injects position information into token embeddings.
Sinusoidal PE (Original Paper)
PE(pos, 2i) = sin(pos / 10000^(2i/dmodel))
PE(pos, 2i+1) = cos(pos / 10000^(2i/dmodel))
pos = token position Β· i = dimension index Β· added directly to embeddings
| PE Type | Learned? | Used In |
|---|---|---|
| Sinusoidal | No β fixed | Original Transformer |
| Learned Absolute | Yes | BERT, GPT-2 |
| Relative (RoPE) | Yes | LLaMA, GPT-NeoX, Mistral |
| ALiBi | No β bias | MPT, BLOOM |
RoPE (Rotary Position Embedding) is the current standard in open LLMs β it encodes relative positions directly in the attention computation via rotation matrices.
Full Transformer Architecture
Encoder Block (ΓN layers)
| Sub-layer | Operation |
|---|---|
| 1. Multi-Head Self-Attention | Each token attends to all tokens |
| 2. Add & LayerNorm | Residual connection + normalise |
| 3. Feed-Forward Network | Two Dense layers: dmodel β 4Β·dmodel β dmodel |
| 4. Add & LayerNorm | Residual connection + normalise |
Decoder Block (ΓN layers)
| Sub-layer | Operation |
|---|---|
| 1. Masked Self-Attention | Attend to past tokens only (causal mask) |
| 2. Add & LayerNorm | Residual + normalise |
| 3. Cross-Attention | Q from decoder, K/V from encoder output |
| 4. Add & LayerNorm | Residual + normalise |
| 5. Feed-Forward + Norm | Same as encoder FFN block |
Feed-Forward Network (FFN)
FFN Formula
FFN(x) = max(0, xWβ+bβ)Wβ+bβ
dmodel=512 β dff=2048 β dmodel=512 Β· applied position-wise independently
Layer Normalisation
LayerNorm vs BatchNorm
LN(x) = Ξ³ Β· (x β ΞΌ) / β(ΟΒ²+Ξ΅) + Ξ²
Normalises across features (not batch) β stable for variable-length sequences
Pre-LN vs Post-LN: Original paper uses Post-LN (after residual). Modern LLMs use Pre-LN (before sub-layer) β more stable training, no learning rate warmup needed.
Transformer Block (Keras)
class TransformerBlock(keras.layers.Layer): def __init__(self, d_model, n_heads, dff): super().__init__() self.attn = keras.layers.MultiHeadAttention( num_heads=n_heads, key_dim=d_model//n_heads) self.ff1 = keras.layers.Dense(dff, activation='relu') self.ff2 = keras.layers.Dense(d_model) self.ln1 = keras.layers.LayerNormalization() self.ln2 = keras.layers.LayerNormalization() def call(self, x, training=False): attn_out = self.attn(x, x) # self-attention x = self.ln1(x + attn_out) # residual + norm ff_out = self.ff2(self.ff1(x)) # FFN return self.ln2(x + ff_out) # residual + norm
BERT
Bidirectional Encoder Representations from Transformers β encoder-only model pretrained on Masked Language Modelling (MLM) and Next Sentence Prediction (NSP).
| BERT-Base | BERT-Large | |
|---|---|---|
| Layers | 12 | 24 |
| d_model | 768 | 1024 |
| Heads | 12 | 16 |
| Parameters | 110M | 340M |
| Max seq len | 512 | 512 |
BERT Pretraining Tasks
# 1. Masked Language Model (MLM) # Mask 15% of tokens β predict them "The [MASK] sat on the mat" # β predict "cat" # 2. Next Sentence Prediction (NSP) # Given sentence A + B, is B next? [CLS] Sentence A [SEP] Sentence B [SEP] # β IsNext or NotNext
[CLS] token: The first token's final hidden state is used as the sequence representation for classification tasks.
GPT Family
Generative Pretrained Transformer β decoder-only, autoregressive model. Trained on next-token prediction with a causal (left-to-right) mask.
| Model | Params | Context | Key Feature |
|---|---|---|---|
| GPT-1 | 117M | 512 | Original GPT β unsupervised pretraining |
| GPT-2 | 1.5B | 1024 | Zero-shot generation; "too dangerous" |
| GPT-3 | 175B | 4096 | Few-shot in-context learning |
| GPT-4 | ~1T? | 128K | Multimodal; RLHF aligned |
| LLaMA 3 | 8Bβ70B | 8K | Open-source; RoPE; GQA |
Causal Language Modelling
# Predict each token from past tokens only # Causal mask: token i cannot attend to j > i P(tokenβ tokenβ, tokenβ, tokenβ, tokenβ) # Training loss: cross-entropy over all positions # L = -Ξ£ log P(xβ x<β)
Transformer Model Zoo
| Model | Type | Pretraining | Best For | Use Via |
|---|---|---|---|---|
| BERT | Encoder | MLM + NSP | Classification, NER, QA | HuggingFace |
| RoBERTa | Encoder | MLM (no NSP, more data) | Better BERT for most NLP | HuggingFace |
| DistilBERT | Encoder | Knowledge distillation of BERT | Fast/lightweight NLP | HuggingFace |
| GPT-2 | Decoder | CLM (causal LM) | Text generation | HuggingFace |
| LLaMA 3 | Decoder | CLM on 15T tokens | Open-source chat/generation | Ollama / HF |
| T5 | Enc-Dec | Span corruption β reconstruct | Translation, summarisation, QA | HuggingFace |
| BART | Enc-Dec | Denoising autoencoder | Summarisation, translation | HuggingFace |
| Whisper | Enc-Dec | Audio β text (multitask) | Speech recognition | OpenAI / HF |
| ViT | Encoder | Image patches as tokens | Image classification | HuggingFace |
| CLIP | Dual Enc | Image-text contrastive | Zero-shot vision-language | OpenAI / HF |
Choosing: Classification/NER β RoBERTa. Generation/chat β LLaMA 3 / GPT. Summarisation/translation β T5 or BART. Speech β Whisper. Vision β ViT or CLIP.
Fine-tuning β HuggingFace
Text Classification (BERT)
from transformers import ( AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer ) # Load pretrained tokenizer = AutoTokenizer.from_pretrained( "bert-base-uncased") model = AutoModelForSequenceClassification\ .from_pretrained("bert-base-uncased", num_labels=2) # Tokenize dataset def tokenize(batch): return tokenizer(batch["text"], padding=True, truncation=True) # Train args = TrainingArguments( output_dir="./results", num_train_epochs=3, per_device_train_batch_size=16, learning_rate=2e-5, weight_decay=0.01, ) trainer = Trainer(model=model, args=args, train_dataset=train_ds, eval_dataset=val_ds) trainer.train()
Fine-tuning Strategies
| Strategy | Updates | GPU RAM | Best For |
|---|---|---|---|
| Full Fine-tune | All params | Very High | Max accuracy; lots of data |
| Head Only | Classifier head | Low | Small datasets; quick baseline |
| LoRA | Low-rank adapters | Low | LLMs on consumer GPU |
| QLoRA | LoRA + 4-bit quant | Very Low | 7Bβ13B models on 1Γ GPU |
| Prompt Tuning | Soft prompt tokens | Minimal | Frozen model; task switching |
| RLHF | Policy + reward model | High | Align to human preferences |
LoRA with PEFT Library
from peft import LoraConfig, get_peft_model config = LoraConfig( r=16, # rank lora_alpha=32, # scaling target_modules=["q_proj", "v_proj"], lora_dropout=0.05, task_type="CAUSAL_LM" ) model = get_peft_model(base_model, config) model.print_trainable_parameters() # trainable params: 4.2M / 6.7B (0.06%)
Tokenisation
| Method | How | Used In |
|---|---|---|
| BPE | Merge frequent byte pairs iteratively | GPT-2, GPT-4, LLaMA, RoBERTa |
| WordPiece | Like BPE but maximises LM likelihood | BERT, DistilBERT |
| SentencePiece | Language-agnostic; treats raw text | T5, LLaMA, mBERT |
| Unigram | Probabilistic; prune vocab by likelihood | XLNet, mBART |
HuggingFace Tokeniser
tokenizer = AutoTokenizer\ .from_pretrained("bert-base-uncased") tokens = tokenizer( "Deep learning is powerful", return_tensors="pt", padding=True, truncation=True, max_length=128 ) # tokens.input_ids, tokens.attention_mask # β [101, 2784, 4083, 2003, 3928, 102]
Token β word: "unbelievable" β ["un", "##believe", "##able"] in WordPiece. Always check your tokeniser's vocabulary and special tokens ([CLS], [SEP], [PAD], [MASK]).
Efficient Attention Variants
| Variant | Complexity | Key Idea |
|---|---|---|
| Full Attention | O(nΒ²) | Baseline β all pairs |
| Sparse Attention | O(nβn) | Only attend to subset of positions |
| Longformer | O(n) | Local window + global tokens |
| Linformer | O(n) | Low-rank K/V projection |
| Flash Attention | O(nΒ²) but IO-aware | Tiled SRAM computation β 2β4Γ faster |
| GQA | O(nΒ²) β fewer KV | Grouped Query Attention β LLaMA 2/3 |
| MQA | O(nΒ²) β 1 KV head | Multi-Query β PaLM, Falcon |
Flash Attention 2 is now the default in most production LLM training β same mathematical result as standard attention but uses hardware-aware tiling to avoid slow HBM reads/writes. Enable with attn_implementation="flash_attention_2" in HuggingFace.
Hyperparameters, Training Tips & Troubleshooting
Key Hyperparameters
| Param | Typical Value |
|---|---|
| Learning rate | 2e-5 β 5e-5 (fine-tune) |
| Batch size | 16 β 64 |
| Warmup steps | 5β10% of total steps |
| Weight decay | 0.01 |
| Max seq length | 128 / 256 / 512 |
| Epochs | 2 β 5 (fine-tune) |
| Dropout | 0.1 (attention + FFN) |
| LoRA rank r | 8 β 64 |
| LR scheduler | Linear decay / cosine |
Common Problems & Fixes
| Problem | Fix |
|---|---|
| Catastrophic forgetting | Lower LR Β· fewer epochs Β· LoRA |
| OOM on GPU | Gradient checkpointing Β· mixed precision Β· QLoRA |
| Loss spikes | LR warmup Β· gradient clipping (max norm 1.0) |
| Slow convergence | Increase warmup Β· check tokenisation |
| Poor generation quality | Tune temperature / top-p / top-k Β· beam search |
| Overfitting small dataset | Early stop Β· reduce epochs Β· data augmentation |
Inference: Generation Settings
HuggingFace generate()
model.generate( input_ids, max_new_tokens=200, do_sample=True, temperature=0.7, # creativity top_p=0.9, # nucleus sampling top_k=50, # top-k sampling repetition_penalty=1.2, num_beams=1 # 1=greedy, >1=beam )
| Setting | Effect |
|---|---|
| temp β 0 | Deterministic (greedy) |
| temp β 1+ | More creative/random |
| top_p=0.9 | Nucleus: top 90% prob mass |
| top_k=50 | Sample from top 50 tokens |
Transformers β Mastery Checklist
Architecture Fundamentals
- Write the scaled dot-product attention formula from memory
- Explain Q, K, V and what each represents intuitively
- Describe why βdβ scaling is used in attention
- Explain multi-head attention and how heads work in parallel
- List all sub-layers in one encoder block and one decoder block
- Explain sinusoidal vs learned vs RoPE positional encoding
- Describe the causal mask used in GPT-style decoders
Pretrained Models
- Distinguish encoder-only, decoder-only, and encoder-decoder models
- Explain BERT's MLM and NSP pretraining objectives
- Explain GPT's causal language modelling objective
- Choose the right model for: classification, generation, summarisation
- Load a pretrained model and tokeniser with HuggingFace AutoModel
- Fine-tune BERT on a classification task using the Trainer API
Efficient Training & Inference
- Explain LoRA and why it reduces trainable parameters
- Apply QLoRA to fine-tune a 7B model on a consumer GPU
- Explain Flash Attention and why it's faster despite same complexity
- Tune temperature, top-p, and top-k for text generation
- Diagnose and fix catastrophic forgetting during fine-tuning
- Enable gradient checkpointing and mixed precision for large models
Deep Learning Series Complete!
All Cheat Sheets β
All 4 Sheets Done β DL Basics Β· CNN Β· RNN & LSTM Β· Transformers
Explore more series: ML Fundamentals Β· Statistics & Math Β· NLP Β· Computer Vision