Deep Learning Basics
layers Β· backprop Β· activation Β· dropout Β· optimizers Β· loss
Sheet 1 of 4
Deep Learning
Beginner β Intermediate
Printable
What Is Deep Learning?
Deep Learning is a subset of Machine Learning that uses multi-layered neural networks (deep neural networks) to learn hierarchical representations of data automatically β without hand-crafted features.
| Concept | Simple Definition |
|---|---|
| Neuron | Basic compute unit: weighted sum + activation |
| Layer | Group of neurons operating in parallel |
| Deep Network | 2+ hidden layers between input and output |
| Parameters | Weights & biases learned during training |
| Epoch | One full pass through the training dataset |
| Batch | Subset of data used per gradient update |
ML vs DL: Traditional ML needs feature engineering; deep learning learns features automatically from raw data (pixels, text, audio).
Network Architecture
A feedforward neural network passes data in one direction β from input to output through hidden layers.
Typical Architecture (Keras)
from tensorflow import keras model = keras.Sequential([ keras.layers.Input(shape=(784,)), # input keras.layers.Dense(256, activation='relu'), # hidden 1 keras.layers.Dense(128, activation='relu'), # hidden 2 keras.layers.Dense(64, activation='relu'), # hidden 3 keras.layers.Dense(10, activation='softmax')# output ])
Neuron Computation
output = activation(W Β· x + b)
W = weight matrix, x = input vector, b = bias term
Rule of thumb: Start with 1β2 hidden layers. Add depth only when underfitting persists after tuning width.
Activation Functions
Activation functions introduce non-linearity, enabling networks to learn complex patterns beyond linear relationships.
| Function | Formula | Use When |
|---|---|---|
| ReLU | max(0, x) | Default for hidden layers; fast & simple |
| Leaky ReLU | max(Ξ±x, x) | Fixes dying ReLU; Ξ±β0.01 |
| Sigmoid | 1/(1+eβ»Λ£) | Binary classification output only |
| Tanh | (eΛ£βeβ»Λ£)/(eΛ£+eβ»Λ£) | RNNs; zero-centered |
| Softmax | eΛ£β± / Ξ£eΛ£Κ² | Multi-class output (probabilities) |
| GELU | xΒ·Ξ¦(x) | Transformers, BERT, GPT |
| Swish | xΒ·Ο(x) | EfficientNet; smooth, bounded below |
Dying ReLU: Neurons can get stuck at 0 output permanently. Use Leaky ReLU, ELU, or careful weight init to mitigate.
Backpropagation
Backprop computes gradients of the loss with respect to each weight by applying the chain rule backwards through the network.
- Forward pass: Compute predictions and loss from input β output.
- Compute loss gradient: βL/βΕ· β how does loss change with output?
- Chain rule backward: Propagate gradient layer by layer using βL/βW = βL/βa Β· βa/βz Β· βz/βW.
- Update weights: W β W β Ξ· Β· βL/βW using optimizer.
- Repeat for each mini-batch until convergence.
Weight Update Rule (SGD)
W β W β Ξ· Β· βL/βW
Ξ· = learning rate Β· βL/βW = gradient of loss w.r.t. weight
Vanishing gradient: Gradients shrink exponentially in deep nets with sigmoid/tanh. Fix: use ReLU, batch norm, residual connections, or gradient clipping.
Loss Functions
| Loss | Task | Keras Name |
|---|---|---|
| MSE | Regression | mse |
| MAE | Regression (robust) | mae |
| Huber | Regression + outliers | huber |
| Binary CE | Binary classification | binary_crossentropy |
| Categorical CE | Multi-class (one-hot) | categorical_crossentropy |
| Sparse CE | Multi-class (int labels) | sparse_categorical_crossentropy |
| KL Divergence | Distribution matching | kl_divergence |
Cross-Entropy Loss
L = βΞ£ yα΅’ Β· log(Ε·α΅’)
yα΅’ = true label Β· Ε·α΅’ = predicted probability
Compile with Loss
model.compile( optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'] )
Optimizers
| Optimizer | Key Idea | Best For |
|---|---|---|
| SGD | Plain gradient descent Β± momentum | CV with tuned LR schedule |
| Momentum | Accumulates velocity in gradient direction | Smooth loss landscapes |
| RMSProp | Adapts LR per parameter via moving avg | RNNs, non-stationary |
| Adam | Momentum + RMSProp; adaptive LR | Default choice β most tasks |
| AdamW | Adam + decoupled weight decay | Transformers, NLP |
| Adagrad | Large LR for rare params | Sparse features, NLP |
Adam Update Rule
mβ = Ξ²βmβββ + (1βΞ²β)gβ
vβ = Ξ²βvβββ + (1βΞ²β)gβΒ²
W β W β Ξ· Β· mΜβ / (βvΜβ + Ξ΅)
Defaults: Ξ²β=0.9 Β· Ξ²β=0.999 Β· Ξ΅=1e-8 Β· Ξ·=0.001
Start with Adam lr=1e-3. Switch to SGD+momentum for fine-tuning vision models β often yields better generalization.
Dropout
Dropout randomly zeroes out neurons during training with probability p, preventing co-adaptation and reducing overfitting.
Dropout in Keras
model = keras.Sequential([ keras.layers.Dense(256, activation='relu'), keras.layers.Dropout(0.3), # drop 30% of neurons keras.layers.Dense(128, activation='relu'), keras.layers.Dropout(0.2), # lower rate near output keras.layers.Dense(10, activation='softmax') ])
| Rate | Typical Usage |
|---|---|
| 0.1β0.2 | Mild regularization, small models |
| 0.3β0.5 | Dense layers in large networks |
| 0.5 | Classic; introduced in original paper |
Disabled at inference: Dropout is active only during training. Keras handles this automatically via model.fit() vs model.predict().
Batch Normalization
Batch Norm normalizes layer inputs across a mini-batch to zero mean and unit variance, then applies learnable scale (Ξ³) and shift (Ξ²).
BatchNorm Formula
xΜ = (x β ΞΌ_B) / β(ΟΒ²_B + Ξ΅)
y = Ξ³ Β· xΜ + Ξ²
ΞΌ_B = batch mean Β· ΟΒ²_B = batch variance Β· Ξ³, Ξ² = learned params
BatchNorm in Keras
keras.layers.Dense(256), keras.layers.BatchNormalization(), # after Dense keras.layers.Activation('relu'), # activation after BN
Benefits
Higher learning rates Β· Reduces sensitivity to init Β· Acts as regularizer Β· Faster convergence
οΈ Caveats
Small batches hurt (use Layer Norm) Β· Extra compute Β· Inference uses running stats
Weight Initialization
Poor initialization causes vanishing or exploding gradients. Use principled schemes that match your activation function.
| Init Method | Formula (std) | Use With |
|---|---|---|
| Xavier / Glorot | β(2/(nα΅’β+nβα΅€β)) | Sigmoid, Tanh |
| He / Kaiming | β(2/nα΅’β) | ReLU, Leaky ReLU |
| LeCun | β(1/nα΅’β) | SELU activation |
| Orthogonal | Orthogonal matrix | RNNs, deep nets |
Set Initializer in Keras
keras.layers.Dense( 128, activation='relu', kernel_initializer='he_normal' # for ReLU )
Never initialize all weights to zero β all neurons learn identically (symmetry problem). Use random init or the methods above.
Training Process
- Prepare data: Split train/val/test Β· Normalize inputs Β· Batch & shuffle.
- Build model: Choose architecture, activations, and init strategy.
- Compile: Select optimizer, loss function, and evaluation metrics.
- Train: model.fit() with epochs, batch size, and validation data.
- Monitor: Plot train vs val loss Β· Watch for overfitting / underfitting.
- Regularize: Add dropout, batch norm, or L2 if overfitting.
- Tune: Adjust learning rate, architecture depth/width, batch size.
- Evaluate: Final metrics on held-out test set β never val set.
Full Training Call
history = model.fit( X_train, y_train, epochs=50, batch_size=32, validation_data=(X_val, y_val), callbacks=[early_stop, lr_scheduler] )
Overfitting vs Underfitting
| Issue | Signs | Fixes |
|---|---|---|
| Overfitting | Low train loss, high val loss | More data Β· Dropout Β· L1/L2 Β· Early stop Β· Smaller model |
| Underfitting | High train + val loss | Bigger model Β· More epochs Β· Lower LR Β· Remove regularization |
| Good Fit | Train β val loss, both low | Model generalizes well β deploy! |
Early Stopping Callback
early_stop = keras.callbacks.EarlyStopping( monitor='val_loss', patience=5, # wait 5 epochs restore_best_weights=True )
Learning curves are your best diagnostic tool. Always plot training and validation loss together across epochs.
Learning Rate Strategies
Learning rate (Ξ·) is the most critical hyperparameter. Too high β diverges. Too low β slow convergence or local minima.
| Strategy | Description |
|---|---|
| Constant | Fixed LR throughout training |
| Step Decay | Reduce by factor every N epochs |
| Exponential Decay | LR Γ e^(βdecayΒ·step) |
| Cosine Annealing | Smooth decay following cosine curve |
| Warm-up + Decay | Ramp up then decay; used in Transformers |
| Cyclical LR | Oscillate between min/max; escapes saddle pts |
| LR Finder | Sweep LR; pick value before loss rises |
ReduceLROnPlateau Callback
lr_scheduler = keras.callbacks.ReduceLROnPlateau( monitor='val_loss', factor=0.5, patience=3, min_lr=1e-6 )
Regularization Techniques β Full Reference
L1 & L2 Weight Penalty
L2 (Ridge) Loss
L_total = L + Ξ» Ξ£ wα΅’Β²
L1 uses wα΅’; L1 drives weights to zero (sparse)
Keras
keras.layers.Dense( 128, kernel_regularizer= keras.regularizers.l2(0.01) )
Data Augmentation
Artificially expand training data by applying label-preserving transformations:
Image Augmentation (Keras)
aug = keras.Sequential([ layers.RandomFlip("horizontal"), layers.RandomRotation(0.1), layers.RandomZoom(0.1), layers.RandomContrast(0.1) ])
NLP: Synonym replacement, back-translation, random masking.
Other Techniques
| Technique | Idea |
|---|---|
| Early Stop | Halt when val loss stops improving |
| Dropout | Random neuron deactivation during train |
| Batch Norm | Implicit regularization via normalization |
| Max-Norm | Clip weight norms to a max value |
| Label Smooth | Soften one-hot targets (0.9 / 0.1) |
| Mixup | Blend two training examples + labels |
Hyperparameter Quick Reference
| Hyperparameter | Typical Range | Notes |
|---|---|---|
| Learning Rate | 1e-4 β 1e-2 | Most critical; start 1e-3 with Adam |
| Batch Size | 16 β 512 | Larger β stable gradients; smaller β regularizes |
| Epochs | 10 β 200+ | Use early stopping; don't fix arbitrarily |
| Hidden Units | 64 β 2048 | Powers of 2 for GPU efficiency |
| Layers (depth) | 2 β 50+ | Start shallow; go deeper if underfitting |
| Dropout Rate | 0.1 β 0.5 | Tune based on train vs val gap |
| L2 Ξ» | 1e-4 β 1e-2 | Too high β underfitting |
Tuning Strategy
- Random search outperforms grid search when β₯3 hyperparameters vary.
- Bayesian optimization (Optuna, Keras Tuner) is most efficient for expensive models.
- Fix architecture first, tune LR and batch size, then regularization.
- Always validate on a separate val set β never the test set.
Keras Tuner (Random Search)
import keras_tuner as kt tuner = kt.RandomSearch( build_model, objective='val_accuracy', max_trials=20 ) tuner.search(X_train, y_train, epochs=10, validation_data=(X_val, y_val))
Key Terminology Glossary
| Term | Meaning |
|---|---|
| Tensor | N-dimensional array (generalized matrix) |
| Gradient | Partial derivative of loss w.r.t. a weight |
| Epoch | One full pass through training data |
| Iteration | One batch update (epoch Γ· batch_size) |
| Inference | Running model on new data (no training) |
| Term | Meaning |
|---|---|
| Forward Pass | Input β output computation |
| Backward Pass | Loss gradient β weights via backprop |
| Overfitting | Memorizes train data; fails on new data |
| Underfitting | Model too simple; misses patterns |
| Checkpoint | Saved model weights at a point in training |
| Term | Meaning |
|---|---|
| Transfer Learning | Reuse pretrained model on new task |
| Fine-tuning | Unfreeze & retrain pretrained layers |
| Flatten | Convert 2D/3D tensor to 1D vector |
| Logits | Raw scores before softmax/sigmoid |
| Convergence | Loss stops decreasing meaningfully |
Deep Learning Basics β Mastery Checklist
Core Concepts
- Explain what a neuron computes (WΒ·x + b + activation)
- Describe the role of each layer type (input, hidden, output)
- Explain why deep networks outperform shallow ones for complex data
- Define epoch, batch, and iteration and how they relate
- Distinguish parameters (weights) from hyperparameters
Training Mechanics
- Trace the forward pass through a 3-layer network by hand
- Explain backpropagation and the chain rule in plain English
- Choose the right loss function for regression vs classification
- Compare Adam, SGD, and RMSProp and when to use each
- Read learning curves and diagnose over/underfitting
- Implement early stopping and LR scheduling callbacks
Regularization & Tuning
- Apply dropout correctly (train only, not inference)
- Insert batch normalization in the right position in a layer stack
- Select He init for ReLU and Xavier for sigmoid/tanh
- Use L1/L2 regularization and tune Ξ»
- Explain when random search beats grid search for HPO
- Run a basic Keras Tuner or Optuna hyperparameter search