Reinforcement Learning Cheat Sheet — Q-Learning, DQN, PPO, Policy Gradients | Dataplexa

Reinforcement Learning Cheat Sheet

MDP · reward · policy · Q-learning · DQN · policy gradients · Actor-Critic · PPO

Sheet 4 of 4 Specialized ML Advanced Printable

Markov Decision Process (MDP)

RL Foundation

Every RL problem is formally defined as an MDP. An agent observes a state, takes an action, receives a reward, and transitions to the next state — repeatedly.

MDP Tuple

MDP = (S, A, P, R, γ)

S = state space · A = action space · P(s'|s,a) = transition probability · R(s,a) = reward function · γ ∈ [0,1) = discount factor

Return (Discounted Cumulative Reward)

G_t = Σ_k=0^∞ γ^k R_t+k+1

γ close to 1 = far-sighted (values future rewards). γ close to 0 = short-sighted (greedy). Discount ensures convergence for infinite horizons.

Element	Symbol	Example (Cart-Pole)
State	s ∈ S	Cart pos, velocity, angle, angular vel
Action	a ∈ A	Push left or push right
Reward	r = R(s,a)	+1 for every step pole stays upright
Transition	P(s'\|s,a)	Physics simulation → next state
Policy	π(a\|s)	Prob of pushing left given current state
Episode	τ	Sequence until pole falls (terminal)

Markov property: The future depends only on the current state, not the full history: P(s_t+1|s_t, a_t) = P(s_t+1|s_0:t, a_0:t). This is the key assumption that makes RL tractable.

State-Value Function V^π

V^π(s) = 𝔼_π[G_t | S_t=s]

Expected return starting from state s, following policy π. Answers: "how good is it to be in state s?"

Action-Value Function Q^π

Q^π(s,a) = 𝔼_π[G_t | S_t=s, A_t=a]

Expected return taking action a in state s, then following policy π. The basis of Q-learning and DQN.

Bellman Equations

Dynamic Programming

Bellman Expectation — V^π

V^π(s) = Σ_a π(a|s) [R(s,a) + γ Σ_s' P(s'|s,a) V^π(s')]

Recursive relationship: value of s = immediate reward + discounted value of next states, averaged over policy and transitions.

Bellman Optimality — Q*

Q*(s,a) = R(s,a) + γ Σ_s' P(s'|s,a) max_a' Q*(s',a')

Optimal Q-value: take action a, then act optimally thereafter. The optimal policy is π*(s) = argmax_a Q*(s,a).

Advantage Function

A^π(s,a) = Q^π(s,a) − V^π(s)

How much better action a is compared to the average action in state s. A > 0 means action is better than baseline. Used in A2C, PPO.

Curse of dimensionality: Tabular methods store V or Q for every (s,a) pair. With large or continuous spaces this is infeasible — we need function approximation (neural networks).

Q-Learning & SARSA

Tabular Methods

Q-Learning Update (Off-Policy)

Q(s,a) ← Q(s,a) + α[r + γ max_a'Q(s',a') − Q(s,a)]

α = learning rate. Learns optimal Q* regardless of the behaviour policy. The bracketed term is the TD error δ.

SARSA Update (On-Policy)

Q(s,a) ← Q(s,a) + α[r + γ Q(s',a') − Q(s,a)]

Uses actual next action a' (from the current policy), not the greedy max. Safer in stochastic environments. Converges to Q^π, not Q*.

Q-Learning — Tabular Python

import numpy as np

Q = np.zeros((n_states, n_actions))
alpha, gamma, eps = 0.1, 0.99, 0.1

for episode in range(1000):
  s = env.reset()
  while True:
    # ε-greedy action selection
    if np.random.rand() < eps:
      a = env.action_space.sample()
    else:
      a = np.argmax(Q[s])
    s2, r, done, _ = env.step(a)
    td = r + gamma*np.max(Q[s2])-Q[s,a]
    Q[s,a] += alpha * td
    s = s2
    if done: break

Deep Q-Network (DQN)

Value-Based Deep RL

DQN replaces the Q-table with a neural network Q(s,a;θ) that generalises across large state spaces (e.g. raw pixels).

DQN Loss (MSE of TD Error)

L(θ) = 𝔼[(r + γ max_a' Q(s',a';θ⁻) − Q(s,a;θ))²]

θ = online network weights. θ⁻ = frozen target network weights. Target network updated every C steps to stabilise training.

Double DQN — Reduces Overestimation

y = r + γ Q(s', argmax_a' Q(s',a';θ); θ⁻)

Action selection with online network θ, value estimation with target θ⁻. Decouples selection from evaluation — reduces upward bias.

DQN Innovation	Problem It Solves
Experience Replay	Breaks correlation between consecutive samples; reuses data
Target Network	Stabilises training by fixing Q-target for C steps
Double DQN	Reduces Q-value overestimation bias
Dueling DQN	Separates V(s) and A(s,a) streams for better generalisation
Prioritised Replay	Samples high TD-error transitions more frequently
Noisy Nets	Learnable noise for exploration instead of ε-greedy

DQN with Stable-Baselines3

from stable_baselines3 import DQN
import gymnasium as gym

env = gym.make("CartPole-v1")

model = DQN(
  "MlpPolicy", env,
  learning_rate=1e-4,
  buffer_size=10_000,
  learning_starts=1000,
  batch_size=32,
  tau=1.0,
  gamma=0.99,
  train_freq=4,
  target_update_interval=1000,
  exploration_fraction=0.1,
  exploration_final_eps=0.05,
  verbose=1
)
model.learn(total_timesteps=50_000)
model.save("dqn_cartpole")

When to use DQN: Discrete action spaces only. For continuous actions (robotics, locomotion), use policy gradient methods (PPO, SAC, TD3) instead.

Policy Gradient Methods

REINFORCE · Actor-Critic

Instead of learning Q-values, policy gradient methods directly optimise the policy π_θ(a|s) by gradient ascent on expected return.

Policy Gradient Theorem

∇_θJ(θ) = 𝔼_π[∇_θ log π_θ(a|s) · Q^π(s,a)]

Log-derivative trick: ∇ log π · Q. Update θ to increase probability of high-return actions. Foundation of REINFORCE and all Actor-Critic methods.

REINFORCE with Baseline

∇_θJ ≈ Σ_t ∇_θ log π_θ(a_t|s_t) (G_t − b(s_t))

b(s) = baseline (usually V(s)) reduces variance without adding bias. G_t − b(s_t) = advantage estimate.

Method	Type	Key Property
REINFORCE	Monte Carlo PG	Simple, high variance
A2C	Advantage AC	Synchronous, stable
A3C	Async AC	Parallel workers, faster
PPO	Clipped PG	Best default, robust
TRPO	Trust region	Theoretically safe updates

PPO — Proximal Policy Optimisation

Default Choice

PPO Clipped Objective

L^CLIP(θ) = 𝔼_t[min(r_t(θ)Â_t, clip(r_t(θ), 1−ε, 1+ε)Â_t)]

r_t(θ) = π_θ(a|s) / π_θold(a|s) = probability ratio. ε = 0.2 typical. Clip prevents large policy updates that destabilise training.

PPO Full Loss

L = L^CLIP − c₁L^VF + c₂S[π_θ]

L^VF = value function MSE loss. S = entropy bonus for exploration. c₁=0.5, c₂=0.01 typical. Single network for actor + critic.

PPO with Stable-Baselines3

from stable_baselines3 import PPO

model = PPO(
  "MlpPolicy", env,
  learning_rate=3e-4,
  n_steps=2048,      # rollout buffer
  batch_size=64,
  n_epochs=10,       # update epochs
  gamma=0.99,
  gae_lambda=0.95,   # GAE smoothing
  clip_range=0.2,    # ε clipping
  ent_coef=0.0,      # entropy coeff
  vf_coef=0.5,       # value loss coeff
  verbose=1
)
model.learn(total_timesteps=1_000_000)

GAE (Generalised Advantage Estimation): λ=0 → TD(0) advantage (low var, high bias). λ=1 → Monte Carlo (high var, low bias). λ=0.95 balances both.

Actor-Critic, SAC & TD3

Continuous Action RL

Actor-Critic methods maintain two networks: the actor π_θ(a|s) selects actions; the critic V_φ(s) or Q_φ(s,a) estimates value and provides a baseline.

Actor Update (Policy Gradient)

∇_θJ = 𝔼[∇_θ log π_θ(a|s) · A(s,a)]

Advantage A(s,a) from critic reduces variance. Actor improves policy. Critic improves value estimates. Both updated simultaneously.

SAC Entropy-Augmented Objective

J(π) = Σ_t 𝔼[r(s_t,a_t) + α H(π(·|s_t))]

α = temperature. H = entropy. Maximising entropy encourages exploration and prevents premature convergence. α auto-tuned in SAC.

Algorithm	Action Space	Key Feature	Best For
A2C	Both	Synchronous, simple	Discrete + continuous
PPO	Both	Clipped ratio, robust	Most tasks — default
SAC	Continuous	Entropy maximisation	Robotics, locomotion
TD3	Continuous	Twin critics + delayed actor	Robotics, high precision
DDPG	Continuous	Deterministic policy	Predecessor to TD3/SAC

SAC — Stable-Baselines3

from stable_baselines3 import SAC
import gymnasium as gym

env = gym.make("Pendulum-v1")
model = SAC(
  "MlpPolicy", env,
  learning_rate=3e-4,
  buffer_size=1_000_000,
  batch_size=256,
  gamma=0.99,
  tau=0.005,
  ent_coef='auto',  # auto-tune α
  verbose=1
)
model.learn(100_000)

TD3 Trick	Why It Helps
Twin Critics	Take min of 2 Q-estimates → reduces overestimation
Delayed Actor	Update policy every 2 critic steps → more stable
Target Noise	Add noise to target actions → smooths Q-landscape

Hyperparameter	Typical Value	Effect
`gamma`	0.99	Discount — horizon length
`tau`	0.005	Soft target update rate
`buffer_size`	1M	Replay memory capacity
`batch_size`	256	SGD mini-batch
`learning_rate`	3e-4	Adam lr for all networks

Exploration Strategies

Exploration vs Exploitation

The agent must balance exploring new actions to discover better rewards vs exploiting known good actions.

ε-Greedy

a = argmax Q(s,a) with prob 1−ε, else random

ε decays from 1.0 → 0.05 over training. Simple but effective for discrete actions.

Upper Confidence Bound (UCB)

a* = argmax_a [Q(a) + c √(ln t / N(a))]

N(a) = times action a was taken. Explores actions with high uncertainty (low N). Optimistic in the face of uncertainty.

Strategy	How	Best For
ε-greedy	Random action with prob ε	DQN, tabular
Boltzmann	Sample ∝ exp(Q/T)	Soft preference
UCB	Optimistic upper bound	Bandits, planning
Entropy bonus	Add H(π) to reward	SAC, PPO
Noisy Nets	Learnable weight noise	Rainbow DQN
Intrinsic reward	Bonus for novel states	Sparse reward envs

Sparse rewards: When rewards are rare (e.g. reaching a goal in a maze), pure ε-greedy fails. Use Hindsight Experience Replay (HER), curiosity-driven exploration, or reward shaping.

Reward Shaping & Design

Reward Engineering

Reward Type	Example	Risk
Sparse	+1 on goal, 0 otherwise	Hard to learn from
Dense	Distance to goal each step	May exploit reward
Shaped	Extra bonus for sub-goals	Reward hacking
Potential-based	φ(s') − φ(s) shaping	Safe — preserves optima
Intrinsic	Surprise / novelty bonus	Agent ignores task

Define the goal clearly — what does success look like? Avoid proxy rewards that can be gamed
Start sparse — if the agent can learn, sparse is cleanest and most reliable
Add shaping carefully — only add dense components if sparse fails; use potential-based shaping to preserve optimal policy
Check for reward hacking — agent will maximise reward, not your intent; watch for unexpected behaviours

Goodhart's Law in RL: "When a measure becomes a target, it ceases to be a good measure." Agents exploit any gap between your reward function and your true objective.

Algorithm Selection & RL Libraries

Practical Guide

Situation	Recommended Algorithm
Discrete actions, small state space	Q-Learning / tabular
Discrete actions, pixel/large state	DQN / Rainbow
Continuous actions, any task	PPO (start here)
Continuous actions, sample efficiency critical	SAC or TD3
Multi-agent environments	MAPPO, QMIX
Sparse rewards, goal-conditioned	HER + SAC/TD3
Sim-to-real robotics	PPO + domain randomisation
LLM alignment / fine-tuning	PPO + RLHF (TRL library)

Library	Best For
`stable-baselines3`	Production RL, all major algos
`gymnasium`	Standard envs (CartPole, Atari, MuJoCo)
`ray[rllib]`	Distributed, large-scale RL
`cleanrl`	Single-file implementations, research
`tianshou`	Modular, PyTorch-native
`trl`	RLHF for LLMs (PPO, DPO)

Full Training Loop with Callbacks

from stable_baselines3 import PPO
from stable_baselines3.common.callbacks import (
  EvalCallback, StopTrainingOnRewardThreshold)
from stable_baselines3.common.monitor import Monitor
import gymnasium as gym

env      = Monitor(gym.make("LunarLander-v2"))
eval_env = Monitor(gym.make("LunarLander-v2"))

# Stop when mean reward >= 200
stop_cb = StopTrainingOnRewardThreshold(
  reward_threshold=200, verbose=1)

eval_cb = EvalCallback(
  eval_env,
  callback_on_new_best=stop_cb,
  eval_freq=5000,
  best_model_save_path="./logs/",
  verbose=1)

model = PPO("MlpPolicy", env,
  learning_rate=3e-4, verbose=1)
model.learn(
  total_timesteps=500_000,
  callback=eval_cb)

# Load and evaluate best model
best = PPO.load("./logs/best_model")
obs, _ = eval_env.reset()
while True:
  action, _ = best.predict(obs)
  obs, r, done, _, _ = eval_env.step(action)
  if done: break

Reinforcement Learning Mastery Checklist

Self-Assessment

Foundations & Value Methods

Define an MDP with S, A, P, R, γ for a real problem
Explain the Markov property and why it matters
Derive the Bellman optimality equation for Q*
Implement tabular Q-learning with ε-greedy exploration
Distinguish on-policy (SARSA) from off-policy (Q-learning)
Explain experience replay and target networks in DQN

Policy Gradient & Actor-Critic

State the policy gradient theorem and its intuition
Explain how a baseline reduces variance in REINFORCE
Describe the PPO clipped objective and why clipping helps
Distinguish GAE λ=0 vs λ=1 and choose an intermediate value
Explain SAC's entropy maximisation and auto-tuned temperature
Name the 3 TD3 tricks and the problem each solves

Exploration, Reward & Practice

Choose the right exploration strategy for discrete vs continuous actions
Design a reward function and check it for Goodhart's Law violations
Apply potential-based reward shaping without changing the optimal policy
Train a PPO agent on LunarLander with EvalCallback
Select the right algorithm given action space and sample budget
Use stable-baselines3 to train, save, and evaluate any RL model

Specialized ML series complete! Explore more series on Dataplexa — Deep Learning (CNN, RNN, Transformers), Statistics & Math, and Language series (Python, JavaScript, Rust).

← Browse All Cheat Sheets

← Back