Calculus for ML
derivatives · partial derivatives · chain rule · gradients · gradient descent · backpropagation · Jacobians
Sheet 4 of 6
Stats & Math
Intermediate
Printable
Derivative Fundamentals
Limit Definition
f'(x) =
lim
h→0
f(x+h) − f(x)
h
Instantaneous rate of change. Slope of the tangent line at point x.
Notation
f'(x) ·
df
dx
· ∂f · Ḋf
Leibniz (df/dx) notation preferred in ML. ∂ for partial derivatives.
Core Differentiation Rules
Power:
d
dx
xn
= nxn−1
Sum: (f+g)' = f' + g'
Product: (fg)' = f'g + fg'
Quotient: (f/g)' = f'g − fg' g2
Sum: (f+g)' = f' + g'
Product: (fg)' = f'g + fg'
Quotient: (f/g)' = f'g − fg' g2
Constant rule: d/dx [c] = 0 · Constant multiple: d/dx [cf] = c·f'
Common Derivatives in ML
f(x) = x² → f'(x) = 2x f(x) = xⁿ → f'(x) = nxⁿ⁻¹ f(x) = eˣ → f'(x) = eˣ f(x) = ln(x) → f'(x) = 1/x f(x) = sin(x) → f'(x) = cos(x) f(x) = cos(x) → f'(x) = -sin(x) Sigmoid σ(x) = 1/(1+e⁻ˣ) σ'(x) = σ(x) · (1 - σ(x)) ReLU: f(x) = max(0, x) f'(x) = 1 if x > 0, else 0
Why it matters in ML: The derivative tells us the direction and rate at which the loss function changes. This is what gradient descent uses to update model weights — we move in the direction of the negative gradient.
Chain Rule
Chain Rule — Single Variable
dy
dx
=
dy
du
·
du
dx
If y = f(u) and u = g(x), then dy/dx = f'(g(x)) · g'(x)
Chain Rule — Worked Example
y = (3x² + 1)⁵ Let u = 3x² + 1 → y = u⁵ dy/du = 5u4 du/dx = 6x dy/dx = 5u4 · 6x = 30x(3x2 + 1)4
Chain Rule in Neural Networks
Output layer → hidden → input ∂L/∂w₁ = ∂L/∂ŷ · ∂ŷ/∂z · ∂z/∂w₁ Each "·" is one application of the chain rule through a layer. This is the core of backprop.
Partial Derivatives
Partial Derivative — Definition
∂f
∂x
=
lim
h→0
f(x+h,y) − f(x,y)
h
Derivative of f with respect to x, treating all other variables as constants.
Example — MSE Loss Partial Derivative
MSE: L(w) = (1/n) Σ(yᵢ - ŷᵢ)² where ŷᵢ = w·xᵢ (linear model) ∂L/∂w = (-2/n) · Σ xᵢ(yᵢ - ŷᵢ) This gradient tells us how much to adjust w to reduce the loss.
Key insight: In a neural network with millions of weights, each weight gets its own partial derivative — telling us exactly how that single weight affects the total loss.
Gradient & Gradient Descent
The Gradient Vector
∇f =
⎡
∂f
∂x₁
,
∂f
∂x₂
, … ,
∂f
∂xₙ
⎤
Vector of all partial derivatives. Points in the direction of steepest increase. Negate it to descend.
Weight Update Rule
w ← w − η · ∇wL
η = learning rate · L = loss function · ∇wL = gradient of loss w.r.t. weights
| GD Variant | Batch size | Trade-off |
|---|---|---|
| Batch GD | All data | Stable but slow, memory-heavy |
| Stochastic (SGD) | 1 sample | Fast but noisy updates |
| Mini-batch GD | 32–256 | Best of both — standard practice |
| Learning Rate η | Effect |
|---|---|
| Too large | Overshoots minimum, may diverge |
| Just right | Converges smoothly to minimum |
| Too small | Very slow convergence, gets stuck |
Gradient Descent — Python
def gradient_descent( X, y, lr=0.01, epochs=100): w = 0; b = 0 n = len(y) for _ in range(epochs): y_hat = w*X + b dw = -2/n * sum(X*(y-y_hat)) db = -2/n * sum(y-y_hat) w -= lr * dw b -= lr * db return w, b
Local vs Global Minima: Gradient descent finds a local minimum, not necessarily the global minimum. In deep networks this is usually fine — most local minima are similarly good. The real risk is saddle points, where the gradient is zero but it's not a minimum.
Backpropagation
Forward & Backward Pass
Forward:
z = Wx + b,
a = σ(z)
Loss: L = loss(a, y)
Backward: ∂L/∂W = ∂L/∂a · ∂a/∂z · ∂z/∂W
Loss: L = loss(a, y)
Backward: ∂L/∂W = ∂L/∂a · ∂a/∂z · ∂z/∂W
Backprop = repeated chain rule, propagated backward through each layer.
Step-by-Step Backprop
Layer: z = wx + b, a = σ(z), L = (a-y)² ∂L/∂a = 2(a - y) ∂a/∂z = σ(z)·(1-σ(z)) # σ'(z) ∂z/∂w = x ∂z/∂b = 1 ∂L/∂w = ∂L/∂a · ∂a/∂z · ∂z/∂w ∂L/∂b = ∂L/∂a · ∂a/∂z · ∂z/∂b
Vanishing gradient problem: In deep networks, multiplying many small σ'(z) values together makes gradients shrink exponentially. Fix: use ReLU activation, batch normalization, or residual connections.
Jacobian & Hessian
Jacobian Matrix
J =
⎡
∂f1/∂x1
···
∂f1/∂xn
⋮ ⋱ ⋮
∂fm/∂x1
···
∂fm/∂xn
⎤
m×n matrix of all first-order partial derivatives for a vector function f: ℝⁿ → ℝᵐ. Used in backprop for vector-to-vector layers.
Hessian Matrix (2nd Derivatives)
Hij =
∂2f
∂xi∂xj
n×n matrix of all second-order partial derivatives. Describes the curvature of the loss surface. Used in second-order optimizers (Newton's method, L-BFGS).
| Matrix | Shape | ML Use |
|---|---|---|
| Jacobian | m × n | Backprop through vector layers |
| Hessian | n × n | Curvature, 2nd-order optimization |
| Gradient | n × 1 | Scalar loss → weight updates |
Advanced Optimizers
Momentum
v ← βv − η∇L
w ← w + v
w ← w + v
β = momentum factor (typically 0.9). Accelerates past flat regions, dampens oscillations. Like a ball rolling downhill.
RMSProp
s ← βs + (1−β)(∇L)2
w ← w − η √s+ε · ∇L
w ← w − η √s+ε · ∇L
Adapts learning rate per parameter. Divides by moving average of squared gradients. Great for RNNs.
Adam (Adaptive Moment Estimation)
m ← β1m + (1−β1)∇L
v ← β2v + (1−β2)(∇L)2
m̂ = m/(1−β1t) v̂ = v/(1−β2t)
w ← w − η m̂ √v̂+ε
v ← β2v + (1−β2)(∇L)2
m̂ = m/(1−β1t) v̂ = v/(1−β2t)
w ← w − η m̂ √v̂+ε
β₁=0.9, β₂=0.999, ε=1e-8 (defaults). Bias-corrected momentum + adaptive learning rate. Default choice for most deep learning.
| Optimizer | Adaptive? | Best For |
|---|---|---|
| SGD | No | Simple models, strong baseline |
| Momentum | No | Faster convergence than SGD |
| RMSProp | Yes | RNNs, non-stationary objectives |
| Adam | Yes ✓ | General default — most models |
| AdamW | Yes ✓ | Adam + weight decay (transformers) |
Rule of thumb: Start with Adam (lr=1e-3). Switch to SGD+momentum for final fine-tuning — it often finds flatter, more generalizable minima.
Activation Function Derivatives
| Function | f(x) | f'(x) |
|---|---|---|
| Sigmoid σ | 1 / (1+e⁻ˣ) | σ(x)·(1−σ(x)) |
| Tanh | (eˣ−e⁻ˣ)/(eˣ+e⁻ˣ) | 1 − tanh²(x) |
| ReLU | max(0, x) | 1 if x > 0, else 0 |
| Leaky ReLU | max(αx, x) | 1 if x > 0, else α |
| ELU | x if x≥0; α(eˣ−1) if x<0 | 1 if x≥0; f(x)+α if x<0 |
| GELU | x·Φ(x) | Φ(x) + x·φ(x) (approx) |
Dying ReLU: Neurons with negative pre-activation output 0 forever — their gradient is always 0, so weights never update. Fix: Leaky ReLU (α=0.01) or ELU keeps a small gradient for negative inputs.
Loss Function Derivatives
Mean Squared Error (Regression)
L =
1
n
Σ(yᵢ − ŷᵢ)2
→
∂L
∂ŷ
=
−2
n
(y−ŷ)
Binary Cross-Entropy (Classification)
L = −[y log(ŷ) + (1−y) log(1−ŷ)]
∂L ∂ŷ = ŷ − y ŷ(1−ŷ)
∂L ∂ŷ = ŷ − y ŷ(1−ŷ)
With sigmoid output: ∂L/∂z = ŷ − y (beautifully simple!).
Cross-entropy + Softmax gradient: ∂L/∂zᵢ = ŷᵢ − yᵢ (predicted − true). This simplification is why cross-entropy is paired with softmax in multi-class classification.
Key Concepts Quick Reference
| Concept | Formula / Rule | ML Use |
|---|---|---|
| Derivative | df/dx | Rate of change of loss |
| Partial ∂ | ∂f/∂xᵢ (hold others fixed) | Per-weight gradient |
| Gradient ∇ | Vector of all partials | Direction to update weights |
| Chain Rule | dy/dx = dy/du · du/dx | Backpropagation |
| Jacobian J | ∂fᵢ/∂xⱼ matrix | Vector layer gradients |
| Hessian H | ∂²f/∂xᵢ∂xⱼ matrix | 2nd-order optimizers |
| GD Concept | Meaning | Typical Value |
|---|---|---|
| η (lr) | Learning rate | 1e-3 to 1e-1 |
| β₁ | 1st moment decay (Adam) | 0.9 |
| β₂ | 2nd moment decay (Adam) | 0.999 |
| ε | Numerical stability | 1e-8 |
| Epoch | One full pass over data | Depends on dataset |
| Batch size | Samples per GD update | 32 – 256 |
Calculus for ML — Mastery Checklist
| Derivatives | Key point |
|---|---|
| Apply power / product / chain rules | d/dx xⁿ = nxⁿ⁻¹ |
| Differentiate sigmoid, tanh, ReLU | σ'= σ(1−σ) |
| Compute partial derivatives | ∂f/∂xᵢ, hold rest |
| Interpret the gradient vector | ∇f points uphill |
| Optimization | Key point |
|---|---|
| Write the GD weight update rule | w ← w − η·∇L |
| Explain mini-batch GD trade-offs | batch 32–256 |
| Describe Adam update steps | β₁=0.9, β₂=0.999 |
| Identify local minima / saddle points | ∇L = 0 |
| Backpropagation | Key point |
|---|---|
| Trace chain rule through a network | ∂L/∂w = chain |
| Compute MSE and BCE gradients | ŷ − y (simple!) |
| Explain vanishing gradient problem | use ReLU / BN |
| Know Jacobian shape for layer | m × n matrix |
Next up → Sheet 5: Looker Studio ·
Metrics, dimensions, data blending, calculated fields, chart types, filters, and sharing reports — a practical reference for Google's BI tool.
Go to Sheet 5 →