RNN & LSTM
sequence Β· gates Β· vanishing gradient Β· GRU Β· bidirectional Β· seq2seq Β· attention
Sheet 3 of 4
Deep Learning
Intermediate
Printable
What Is an RNN?
A Recurrent Neural Network (RNN) processes sequential data by maintaining a hidden state that carries information from previous time steps β giving the network a form of memory.
Vanilla RNN Equations
hβ = tanh(Wβ Β· hβββ + Wβ Β· xβ + b)
Ε·β = Wᡧ Β· hβ + bᡧ
hβ = hidden state Β· xβ = input Β· Ε·β = output Β· W = weight matrices
| RNN Mode | Input β Output | Example Use |
|---|---|---|
| One-to-One | 1 β 1 | Standard classification |
| One-to-Many | 1 β sequence | Image captioning |
| Many-to-One | Sequence β 1 | Sentiment analysis |
| Many-to-Many (sync) | Seq β Seq (same len) | POS tagging |
| Many-to-Many (async) | Seq β Seq (diff len) | Machine translation |
Vanishing Gradient Problem
During backpropagation through time (BPTT), gradients are multiplied by weight matrices at each step. With long sequences, gradients shrink exponentially β early time steps receive near-zero updates.
Gradient Magnitude Over T Steps
βL/βhβ = βα΅’ (βhα΅’/βhα΅’ββ) Β· βL/βhα΅
If βhα΅’/βhα΅’ββ < 1 repeatedly β gradient β 0 (vanishing)
| Problem | Fix |
|---|---|
| Vanishing gradient | LSTM / GRU gating Β· ReLU activations Β· gradient clipping |
| Exploding gradient | Gradient clipping Β· weight regularization |
| Long-range dependencies | LSTM Β· Attention Β· Transformers |
| Slow training | Truncated BPTT Β· smaller sequence chunks |
Gradient Clipping (Keras)
optimizer = keras.optimizers.Adam( learning_rate=1e-3, clipnorm=1.0 # clip gradient norm to 1 )
LSTM β Long Short-Term Memory
LSTM introduces a cell state (Cβ) β a conveyor belt running through the sequence β protected by three learnable gates that control information flow.
LSTM Gate Equations
Forget fβ = Ο(WfΒ·[hβββ, xβ] + bf)
Input iβ = Ο(WiΒ·[hβββ, xβ] + bi)
Cell CΜβ = tanh(WcΒ·[hβββ, xβ] + bc)
Output oβ = Ο(WoΒ·[hβββ, xβ] + bo)
Cβ = fβ β Cβββ + iβ β CΜβ
hβ = oβ β tanh(Cβ)
Ο = sigmoid (0β1 gate) Β· tanh = cell/hidden Β· β = element-wise multiply
What Each Gate Does
| Gate | Controls | Value |
|---|---|---|
| Forget (fβ) | How much of old cell state Cβββ to keep | 0 = forget all Β· 1 = keep all |
| Input (iβ) | How much new info CΜβ to write to cell | 0 = write nothing Β· 1 = write all |
| Cell (CΜβ) | Candidate new info from current input | β1 to +1 (tanh range) |
| Output (oβ) | How much cell state to expose as hidden hβ | 0 = hide Β· 1 = expose |
LSTM in Keras
# Return last hidden state only keras.layers.LSTM(128) # Return all hidden states (for seq2seq) keras.layers.LSTM(128, return_sequences=True) # Return state + sequences keras.layers.LSTM(128, return_sequences=True, return_state=True) # Stack two LSTM layers keras.layers.LSTM(128, return_sequences=True), keras.layers.LSTM(64)
GRU β Gated Recurrent Unit
GRU merges the forget and input gates into a single update gate and removes the separate cell state β simpler, faster, often matches LSTM performance.
GRU Equations
Reset rβ = Ο(WrΒ·[hβββ, xβ])
Update zβ = Ο(WzΒ·[hβββ, xβ])
New Γ±β = tanh(WnΒ·[rββhβββ, xβ])
hβ = (1βzβ)βhβββ + zββΓ±β
No separate cell state β hidden state hβ carries everything
| LSTM | GRU | |
|---|---|---|
| Gates | 3 (forget, input, output) | 2 (reset, update) |
| States | Cell state + hidden state | Hidden state only |
| Parameters | More (~4Γ hiddenΒ²) | Fewer (~3Γ hiddenΒ²) |
| Speed | Slower | ~33% faster |
| Best for | Long dependencies; NLP | Shorter seq; less data |
GRU in Keras
keras.layers.GRU(128, return_sequences=True) keras.layers.GRU(64)
Bidirectional RNNs
A Bidirectional RNN runs two RNNs β one forward (leftβright) and one backward (rightβleft) β and concatenates their outputs at each step, giving each position context from both directions.
| Direction | Sees | Used In |
|---|---|---|
| Forward β | Past context only | Language generation (GPT style) |
| Backward β | Future context only | β |
| Bidirectional | Past + future | NER, POS tagging, BERT encoder |
Bidirectional LSTM (Keras)
# Wraps any RNN layer keras.layers.Bidirectional( keras.layers.LSTM(64, return_sequences=True) ) # Output size = 2 Γ 64 = 128 (concat) # merge_mode options: # 'concat' (default) 'sum' 'mul' 'ave' keras.layers.Bidirectional( keras.layers.LSTM(64), merge_mode='sum' # β output size = 64 )
Use bidirectional when the full sequence is available at inference time (classification, translation encoder). Avoid for real-time/streaming where future tokens aren't yet known.
Input Preparation
Tokenize & Pad Text Sequences
# Tokenize tokenizer = keras.preprocessing.text.Tokenizer( num_words=10000, oov_token="<OOV>" ) tokenizer.fit_on_texts(train_texts) seqs = tokenizer.texts_to_sequences(train_texts) # Pad to fixed length X = keras.preprocessing.sequence.pad_sequences( seqs, maxlen=200, padding='post', truncating='post' ) # Shape: (num_samples, 200)
| RNN Input Shape | Meaning |
|---|---|
| (batch, timesteps, features) | 3D tensor required by all Keras RNN layers |
| (32, 200, 1) | 32 samples Β· 200 time steps Β· 1 feature each |
| (32, 200, 128) | 32 samples Β· 200 tokens Β· 128-dim embedding |
Masking: Use keras.layers.Masking(mask_value=0) before your RNN to tell Keras which timesteps are padding β the RNN will ignore them during training.
Embeddings
An Embedding layer maps integer token IDs to dense vectors β learnable lookup table of shape (vocab_size, embed_dim).
Embedding Layer (Keras)
# Learned from scratch keras.layers.Embedding( input_dim=10000, # vocab size output_dim=128, # embedding dimension input_length=200, # sequence length mask_zero=True # auto-mask padding ) # Pretrained GloVe / Word2Vec weights keras.layers.Embedding( vocab_size, 300, weights=[embedding_matrix], trainable=False # freeze pretrained )
| Embedding | Dim | Vocab |
|---|---|---|
| GloVe 6B | 50 / 100 / 200 / 300 | 400K words |
| Word2Vec | 300 | 3M words |
| FastText | 300 | Handles OOV via subwords |
| Learned | 64β256 | Task-specific vocab |
Sequence-to-Sequence (Seq2Seq)
Seq2Seq uses an encoder to compress the input sequence into a context vector (final hidden state), then a decoder generates the output sequence one token at a time using that context.
| Component | Role |
|---|---|
| Encoder | Reads input sequence β context vector (hβ, Cβ) |
| Context vector | Final encoder hidden + cell state |
| Decoder | Generates output seq token-by-token, conditioned on context |
| Teacher forcing | Feed true output as decoder input during training |
| Greedy decode | At inference, feed predicted token as next input |
| Beam search | Keep top-k candidates at each step β better quality |
Bottleneck problem: All source information is compressed into one fixed-size vector β hurts long sequences. Solution: Attention mechanism.
Seq2Seq Encoder (Keras)
# Encoder enc_inputs = keras.Input(shape=(None,)) enc_emb = keras.layers.Embedding( src_vocab, 256)(enc_inputs) enc_out, h, c = keras.layers.LSTM( 256, return_state=True)(enc_emb) enc_states = [h, c] # Decoder dec_inputs = keras.Input(shape=(None,)) dec_emb = keras.layers.Embedding( tgt_vocab, 256)(dec_inputs) dec_lstm = keras.layers.LSTM( 256, return_sequences=True, return_state=True) dec_out, _, _ = dec_lstm( dec_emb, initial_state=enc_states) dec_dense = keras.layers.Dense( tgt_vocab, activation='softmax') outputs = dec_dense(dec_out)
Attention Mechanism
Attention lets the decoder look back at all encoder outputs at each decoding step β computing a weighted sum to focus on the most relevant input positions.
Scaled Dot-Product Attention
Attention(Q,K,V) = softmax(QKα΅/βdβ)Β·V
Q = query Β· K = key Β· V = value Β· dβ = key dimension (scaling factor)
| Type | Description |
|---|---|
| Bahdanau (Additive) | Score = Vα΅Β·tanh(WβΒ·hβ + WβΒ·hβ) β original attention |
| Luong (Multiplicative) | Score = hβα΅Β·WβΒ·hβ β simpler, faster |
| Scaled Dot-Product | Score = QKα΅/βdβ β Transformer standard |
| Multi-Head | Run attention h times in parallel, concat β Transformers |
| Self-Attention | Q, K, V all from same sequence β BERT, GPT |
Attention β Transformers: When you apply self-attention without any RNN, stacked with positional encoding and feed-forward layers, you get a Transformer β see Sheet 4.
Practical RNN Patterns
Sentiment Classifier (Many-to-One)
model = keras.Sequential([ keras.layers.Embedding(10000, 128, mask_zero=True), keras.layers.Bidirectional( keras.layers.LSTM(64)), keras.layers.Dense(64, activation='relu'), keras.layers.Dropout(0.4), keras.layers.Dense(1, activation='sigmoid') ])
Time Series Forecasting (Many-to-One)
model = keras.Sequential([ keras.layers.LSTM(64, return_sequences=True, input_shape=(30, 1)), # 30 steps, 1 feature keras.layers.LSTM(32), keras.layers.Dense(1) # predict next value ])
Stateful RNNs: Set stateful=True to carry hidden state across batches β useful for very long time series where you split into chunks.
Hyperparameters, Tips & Troubleshooting
Key Hyperparameters
| Param | Typical Range |
|---|---|
| Hidden units | 64 β 512 |
| Num layers | 1 β 4 stacked |
| Dropout | 0.2 β 0.5 |
| Recurrent dropout | 0.1 β 0.3 |
| Embedding dim | 64 β 300 |
| Sequence length | 50 β 500 (NLP) |
| Batch size | 32 β 128 |
| Learning rate | 1e-3 (Adam) |
| Gradient clip | clipnorm=1.0 |
Common Problems & Fixes
| Problem | Fix |
|---|---|
| Loss not decreasing | Gradient clipping Β· lower LR Β· check input shape |
| Overfitting | Dropout Β· recurrent_dropout Β· smaller model |
| Forgetting long context | Use LSTM or GRU instead of vanilla RNN |
| Slow training | Use CuDNN-compatible (no recurrent_dropout on GPU) |
| NaN loss | Gradient clipping Β· lower LR Β· check embeddings |
| Poor seq2seq quality | Add attention Β· use beam search Β· more data |
When to Use What
| Task | Best Choice |
|---|---|
| Sentiment analysis | BiLSTM or BERT |
| Time series forecast | LSTM / GRU |
| Machine translation | Transformer (Seq2Seq + attention) |
| Named entity recognition | BiLSTM-CRF or BERT |
| Speech recognition | BiLSTM + CTC loss |
| Text generation | LSTM or GPT-style Transformer |
| Short sequences (<50) | GRU (faster, fewer params) |
| Long sequences (>200) | Transformer (no vanishing gradient) |
RNN vs LSTM vs GRU vs Transformer
| Model | Long-Range Deps | Speed | Parallelizable | Memory | Best For |
|---|---|---|---|---|---|
| Vanilla RNN | No Poor | Fast | No Sequential | Low | Very short sequences only |
| LSTM | Good | Medium | No Sequential | Medium | NLP, time series, audio |
| GRU | Good | Faster than LSTM | No Sequential | LowβMed | When LSTM is too slow/large |
| BiLSTM | Best RNN | Slowest RNN | No Sequential | High | NER, encoding, classification |
| Transformer | Global | Slow train, fast inf | Full parallel | Very High | NLP SOTA β BERT, GPT, T5 |
2024 reality: For most NLP tasks, pretrained Transformers (BERT, GPT-2, T5) outperform LSTM-based models significantly. Use LSTM/GRU for time series, resource-constrained environments, or when interpretability of recurrence matters.
RNN & LSTM β Mastery Checklist
RNN Foundations
- Write the vanilla RNN equation hβ = tanh(Wβhβββ + Wβxβ + b) from memory
- Explain the 5 RNN modes (one-to-one through many-to-many)
- Explain why vanishing gradients occur during BPTT
- Apply gradient clipping with clipnorm in Keras
- Prepare and pad a text sequence for RNN input
- Shape a 3D input tensor (batch, timesteps, features) correctly
LSTM & GRU
- Name all 4 LSTM gates and what each controls
- Trace the cell state update: Cβ = fββCβββ + iββCΜβ
- Compare GRU vs LSTM β gates, states, speed, use case
- Stack two LSTM layers using return_sequences=True
- Wrap an LSTM with Bidirectional and explain output size
- Choose between LSTM and GRU for a given task and dataset size
Seq2Seq & Attention
- Describe the encoder-decoder architecture and context vector
- Explain the bottleneck problem in vanilla seq2seq
- Write the scaled dot-product attention formula
- Distinguish self-attention from cross-attention
- Explain when to use BiLSTM vs Transformer for NLP
- Build a sentiment classifier with Embedding + BiLSTM in Keras