Reinforcement Learning

4,900ELO — AlphaGo vs human

40%Cooling energy saved (DeepMind DC)

PPOAlgorithm powering RLHF

2016AlphaGo beats Lee Sedol

00

What is Reinforcement Learning?

RL is a learning paradigm where an agent learns to make decisions by interacting with an environment. Unlike supervised learning (learn from labelled examples), RL learns by trial and error — taking actions, receiving reward signals, and improving over time. No teacher says "do this"; only reward/punishment from the environment guides learning. Like learning to ride a bike: fall → penalty, balance → reward, practice → mastery.

Supervised ML: learn from labels · RL: learn from interaction + reward signals

01

The Agent–Environment Loop

At each timestep: (1) Agent observes state sₜ. (2) Agent selects action aₜ using its policy π. (3) Environment transitions to new state sₜ₊₁ and emits reward rₜ. (4) Agent updates its policy to get more reward. This loop repeats until the episode ends or the agent masters the task. Every RL problem reduces to this interaction loop.

Agent → Action aₜ → Environment → (State sₜ₊₁, Reward rₜ) → Agent (repeat)

02

Policy π — The Agent's Brain

The policy maps states to actions — it's what the agent is trying to learn. Deterministic policy: a = π(s) — always the same action in a given state. Stochastic policy: a ~ π(a|s) — probability distribution over actions. Stochastic policies are often better — they maintain exploration and are more robust. The entire goal of RL is to find the optimal policy π* that maximises expected return.

Deterministic: a = π(s) · Stochastic: P(a|s) · Optimal: π* = argmax_π E[G]

03

Reward & Discounted Return

Reward rₜ: immediate scalar feedback from the environment at step t. Return Gₜ: sum of all future rewards from step t onwards. Discount factor γ∈[0,1]: weights near-future rewards more than distant ones. γ=1: care equally about all future rewards. γ=0: care only about immediate reward. γ=0.99 is typical in practice — values rewards near equally but discounts exponentially far-future ones.

Gₜ = rₜ + γrₜ₊₁ + γ²rₜ₊₂ + ... = Σₖ₌₀^∞ γᵏrₜ₊ₖ · γ=0.99 typical

04

Exploration vs Exploitation

The fundamental RL dilemma. Exploitation: use your best known action — maximise immediate reward based on current knowledge. Exploration: try something new — you might discover better actions. Too much exploitation: stuck in suboptimal behaviour. Too much exploration: waste time on known bad actions. ε-greedy: explore randomly with prob ε, exploit otherwise. ε decays from 1.0 → 0.01 as training progresses.

ε-greedy: if rand() < ε → explore (random) · else → exploit (argmax Q) · ε: 1.0 → 0.01

05

RL vs Supervised vs Unsupervised

Supervised ML: given (X, y) pairs — learn the mapping. Unsupervised ML: given X only — find structure. RL: no dataset given — agent generates its own experience through interaction. Reward signal is delayed (may not know which action caused success), sparse (reward only at game end), and non-stationary (as policy improves, the data distribution changes). These properties make RL fundamentally harder than supervised learning.

RL challenges: delayed reward · sparse signal · non-stationary distribution · no i.i.d. data

Agent

Policy π

→

action aₜ

Environment

State transitions

→

sₜ₊₁, rₜ

Agent

Update policy

00

Markov Decision Process (MDP)

The formal mathematical framework for RL. An MDP is defined by the tuple (S, A, P, R, γ): S — state space (all possible states). A — action space (all possible actions). P(s'|s,a) — transition probability: probability of reaching state s' from state s with action a. R(s,a) — reward function. γ — discount factor. If you know P and R completely, you have a "model" and can use dynamic programming. In most real problems, P is unknown — model-free RL.

MDP = (S, A, P, R, γ) · P known → model-based · P unknown → model-free RL

01

The Markov Property

The future depends only on the current state, not the full history. P(sₜ₊₁|sₜ,aₜ) = P(sₜ₊₁|s₀,...,sₜ,a₀,...,aₜ). "The present captures everything relevant about the past." This simplification is what makes MDPs tractable. In practice, many environments aren't truly Markov — use frame stacking (Atari) or recurrent networks (LSTMs) to include history when needed.

P(s'|s,a) — next state depends only on current state, not history · "memory-less"

02

Value Functions — V(s) and Q(s,a)

State value V^π(s): expected discounted return starting from state s, following policy π. Tells you "how good is it to be in state s?" Action value Q^π(s,a): expected return from state s, taking action a, then following π. More useful — directly tells you which action to take. Advantage A(s,a) = Q(s,a) − V(s): how much better is action a than average — used in actor-critic methods.

V^π(s) = E[Gₜ|sₜ=s] · Q^π(s,a) = E[Gₜ|sₜ=s,aₜ=a] · A(s,a) = Q(s,a)-V(s)

03

Bellman Equations

Recursive decomposition: the value of a state equals the immediate reward plus the discounted value of the next state. This bootstrapping relationship is the foundation for all value-based RL algorithms. Bellman optimality: optimal policy always takes the action that maximises Q*. If we know Q*, we're done — just act greedily.

Q*(s,a) = R(s,a) + γ · Σ_{s'} P(s'|s,a) · max_{a'} Q*(s',a') · Bellman optimality

04

Partial Observability (POMDP)

In a Partially Observable MDP, the agent doesn't see the full state — only a noisy observation. Real-world examples: a robot can't see what's behind it; an Atari game shows only the current frame (is a ball moving left or right?). Solutions: frame stacking (use last 4 frames), recurrent policies (LSTM maintains memory), belief states (probability distribution over states).

POMDP: observe oₜ ≠ sₜ · fix: stack frames · LSTM · belief state b(s) = P(s|observations)

Algorithm Taxonomy

Dynamic Programming

Value-Based

Requires complete model (P, R known). Policy iteration or value iteration to find V* or Q*. Only works for small, fully known environments.

V*(s) = max_a [R + γΣP(s'|s,a)V*(s')]

Q-Learning

Off-Policy TD

Model-free. Learns Q* directly from experience. Off-policy: learns from any behaviour. Foundation for DQN. Tabular: only for small discrete spaces.

Q(s,a) ← Q(s,a) + α[r + γ·max_a' Q(s',a') - Q(s,a)]

SARSA

On-Policy TD

Like Q-learning but updates using the actual next action taken (on-policy). More conservative. Better for risky environments where exploration could be costly.

Q(s,a) ← Q(s,a) + α[r + γ·Q(s',a') - Q(s,a)]

Monte Carlo

On-Policy

Learn from complete episode returns. No bootstrapping. High variance but unbiased. Works for episodic tasks. Can't learn online mid-episode.

V(s) ← average(returns from visits to s)

DQN

Deep Value

Q-learning + neural network. Experience replay + target network for stability. Solved Atari from pixels. The bridge from tabular to deep RL.

Q(s,a;θ) ← r + γ·max_a' Q(s',a';θ⁻)

PPO

Policy Gradient

Proximal Policy Optimisation. Clips policy updates for stability. Simple, robust, widely used. Powers RLHF for LLMs, robotics, game AI. OpenAI's default.

L^CLIP = E[min(rA, clip(r,1-ε,1+ε)A)]

SAC

Off-Policy PG

Soft Actor-Critic. Maximum entropy RL: maximise reward + randomness (exploration). Off-policy: sample efficient. State-of-art for continuous control (robotics).

J(π) = E[Σ rₜ + α·H(π(·|sₜ))]

TD3

Off-Policy PG

Twin Delayed DDPG. Uses two critics (reduces overestimation), delayed policy updates, target policy smoothing. Excellent for continuous control tasks.

min(Q₁, Q₂) target · delayed actor update

Model-Free vs Model-Based

Property	Model-Free	Model-Based
Knows P(s'\|s,a)?	No — learns from interaction	Yes — or learns a model
Sample efficiency	Low — needs many episodes	High — plans from model
Complexity	Simple	Higher — model learning adds error
Algorithms	Q-learning, PPO, SAC, DQN	Dyna-Q, MCTS, World Models, Dreamer
Best for	Games, LLM alignment, unknown dynamics	Planning, sim-to-real, robot manipulation

01

DQN — Deep Q-Network (DeepMind, 2013)

Replace the Q-table with a neural network: input = game screen pixels (84×84×4 grayscale frames stacked), output = Q(s,a) for all actions. Two critical tricks: Experience replay: store (s,a,r,s') in a buffer, sample random minibatches — breaks temporal correlation. Target network: separate frozen network θ⁻ for TD targets, updated every N steps — prevents oscillation. Achieved human-level performance on 49 Atari games from raw pixels.

Q(s,a;θ) ← r + γ·max_{a'} Q(s',a';θ⁻) · replay buffer · target network updated every 10k steps

02

Double DQN & Dueling DQN

Double DQN: DQN overestimates Q-values because it uses the same network to select and evaluate actions. Fix: use online network to select action, target network to evaluate. Reduces overestimation bias significantly. Dueling DQN: decompose Q(s,a) = V(s) + A(s,a) — value of being in state (shared) plus advantage of each action. Better generalisation: learns V(s) even for states where some actions are irrelevant.

Double: select a* = argmax_a Q(s',a;θ) · evaluate Q(s',a*;θ⁻) · Dueling: Q=V+A-mean(A)

03

Actor-Critic Architecture

Combines value-based and policy gradient approaches. Actor π(a|s;θ): the policy network — outputs action probabilities. Critic V(s;φ) or Q(s,a;φ): the value network — estimates return. Actor updates its policy using the critic's advantage estimate: how much better is this action than average? Critic updates via standard TD. Lower variance than pure policy gradient (which must wait for episode end).

Actor: θ ← θ + α·∇log π(a|s;θ)·A(s,a) · Critic: φ ← φ + β·TD error · run simultaneously

04

A3C / A2C — Parallel Actors

A3C (Asynchronous): multiple workers run independently in parallel, asynchronously updating a global network. Naturally decorrelates experience. Fast but noisy gradients. A2C (Synchronous): workers collect experience simultaneously, then all update together. More stable, comparable performance, simpler to implement. A2C is now generally preferred. Both use entropy bonus to encourage exploration.

A2C: N workers → synchronous rollouts → global update · faster than serial + decorrelated

05

AlphaGo / AlphaZero — The MCTS + Deep RL Combination

DeepMind combined Monte Carlo Tree Search with deep RL. Policy network: suggests promising moves (trained by RL). Value network: estimates win probability (trained by RL). MCTS uses both to guide simulated playouts. AlphaZero: starts from zero knowledge, trains purely through self-play — no human games, no handcrafted features. Mastered Chess, Shogi, and Go simultaneously to superhuman level in hours.

MCTS × policy_net(board) × value_net(board) → best move · self-play → superhuman in days

00

Why Policy Gradient?

Value-based methods (Q-learning) select actions greedily from Q-values — but this requires discrete action spaces and can't handle continuous actions (robot joint angles, steering). Policy gradient methods directly optimise the policy network using gradient ascent on expected return. Work naturally with continuous action spaces, stochastic policies, and partial observability. The backbone of modern deep RL and RLHF.

Q-learning → discrete actions only · Policy gradient → continuous + stochastic actions

01

Policy Gradient Theorem

The gradient of expected return with respect to policy parameters. The "log trick" (∇log π = ∇π/π) converts the gradient of a probability into something computable from sampled trajectories. Intuition: increase the log-probability of actions that led to high return; decrease it for low-return actions. The Q-value (or advantage) weights how much to push each action.

∇_θ J(θ) = E_π[∇_θ log π(a|s;θ) · Q^π(s,a)] · increase prob of good actions

02

REINFORCE — Monte Carlo Policy Gradient

Simplest policy gradient: run a full episode, use discounted return Gₜ as the Q estimate, update parameters. Simple but extremely high variance — return from a single trajectory is noisy. Baseline trick: subtract V(s) from return to get advantage A(s,a) = Gₜ − V(sₜ). Same gradient direction in expectation but dramatically lower variance. The baseline doesn't bias the gradient.

θ ← θ + α·∇_θ log π(aₜ|sₜ;θ)·(Gₜ - V(sₜ)) · baseline V(s) reduces variance

03

PPO — Proximal Policy Optimisation

OpenAI's workhorse algorithm. The key insight: large policy updates can be catastrophically destabilising — the new policy might be so different that the value estimates become meaningless. Fix: clip the probability ratio r = π_new(a|s) / π_old(a|s) to stay within [1−ε, 1+ε]. This prevents any single update from changing the policy too drastically. Simple, robust, widely applicable. Powers robotics, game AI, and RLHF for LLMs.

L^CLIP(θ) = E[min(r(θ)·A, clip(r(θ), 1-ε, 1+ε)·A)] · ε=0.1 or 0.2 typical

04

SAC — Soft Actor-Critic

Maximum entropy RL: augment the reward with the policy's entropy H(π(·|s)). This encourages the policy to be as random as possible while still maximising reward. Benefits: automatic exploration without ε-greedy, robustness (multiple near-optimal actions reinforced), better generalisation. Off-policy: uses a replay buffer — extremely sample efficient. State-of-art for continuous control tasks.

J(π) = E[Σ rₜ + α·H(π(·|sₜ))] · α = temperature (entropy weight) · off-policy = efficient

00

What is RLHF?

Reinforcement Learning from Human Feedback aligns large language models with human values and preferences. The key challenge: how do you define a reward function for "helpfulness" or "harmlessness"? You can't write it as a formula. RLHF replaces a hand-crafted reward with human preference judgements, learned into a reward model. Used to produce ChatGPT, Claude, Gemini, and every major conversational AI.

Problem: can't write reward(helpful) as formula · Solution: learn reward from human preferences

01

Step 1 — Supervised Fine-Tuning (SFT)

Start from a pretrained LLM. Fine-tune on high-quality (prompt, ideal response) pairs written by human contractors. Teaches the base model to behave as an assistant — follow instructions, maintain helpful tone, format answers correctly. Typically 10k–100k curated examples. Without SFT, the base model generates incoherent text; with SFT, it's a capable but potentially unsafe assistant.

Base LLM → SFT on {prompt, ideal_response} pairs → instruction-following model

02

Step 2 — Reward Model Training

Human raters compare pairs of model responses: "Response A or B — which is better?" Collect hundreds of thousands of such preference pairs. Train a separate reward model (same architecture as SFT model, regression head) to predict which response humans would prefer. This reward model becomes the automated "human judge" during RL training — much cheaper than using real humans every step.

RM(prompt, response_A, response_B) → P(A preferred) · trained on human preference pairs

03

Step 3 — PPO Optimisation

Use PPO to fine-tune the SFT model to maximise reward model scores. Critical stabilisation: add a KL divergence penalty between the RL-optimised policy and the SFT model — prevents the model from "gaming" the reward model by producing gibberish that gets high scores. The KL penalty keeps the model close to the SFT baseline while improving along the reward dimension.

reward = RM(response) - β·KL(π_RL || π_SFT) · β prevents reward hacking + mode collapse

04

DPO — Direct Preference Optimisation (2023)

A simpler alternative that skips the reward model and PPO entirely. Directly optimises on preference pairs (chosen > rejected) using a supervised loss. The key insight: the optimal policy under RLHF with KL constraint has a closed-form solution — you can derive a loss that optimises it directly without ever running RL. Same performance as RLHF, more stable training, no reward model needed. Now widely preferred in open-source community (used by Llama 3, Mistral, Zephyr).

L_DPO = -log σ(β[log π(y_w|x) - log π(y_l|x) - same for ref model]) · no RM, no PPO

Reward hacking: RL systems will find unexpected ways to maximise the reward function that violate the intent. An RL boat racing agent found it could score higher by driving in circles collecting power-ups than finishing the race. The KL penalty in RLHF prevents LLMs from exploiting the reward model with incoherent high-scoring outputs.

Game Playing

AlphaGo/Zero (Go), OpenAI Five (Dota 2 — 5v5 at pro level), AlphaStar (StarCraft II Grandmaster), DQN (49 Atari games from pixels).

DQN · AlphaZero · PPO · MCTS+RL

Robotics

Locomotion (bipedal walking, backflips), dexterous manipulation, robotic surgery. Google DeepMind RT-2: language-conditioned robot actions.

SAC · TD3 · PPO · sim-to-real transfer

LLM Alignment

RLHF makes ChatGPT, Claude, Gemini helpful, harmless, honest. DPO increasingly preferred. Constitutional AI (Anthropic) uses RL with AI feedback.

PPO + KL penalty · DPO · CAI

Resource Management

DeepMind's RL agent reduced Google datacenter cooling energy by 40%. RL for CPU/GPU scheduling, network routing, cloud resource allocation.

PPO · Q-learning · custom envs

Finance & Trading

Algorithmic trading, portfolio optimisation, order execution to minimise market impact. J.P. Morgan, Goldman Sachs use RL for execution strategies.

DQN · SAC · multi-agent RL

Autonomous Driving

RL for high-level decision making in complex traffic scenarios. Waymo uses RL in simulation. Tesla uses imitation learning + RL for FSD.

PPO + MCTS · sim-to-real · safety constraints

Drug Discovery

RL for molecular design — generate molecules with desired properties (high binding affinity, low toxicity). Treats molecule generation as a sequential decision process.

Policy gradient · REINFORCE · molecular graphs

Nuclear Fusion

DeepMind (2022): RL agent controls plasma shape in a tokamak reactor. Maintained complex plasma configurations that previously required extensive manual tuning.

Custom RL · continuous control · safety

python — Q-Learning on CartPole with ε-greedy exploration

import gymnasium as gym
import numpy as np

env = gym.make('CartPole-v1')
Q   = np.zeros((10**4, env.action_space.n))

alpha, gamma, epsilon = 0.1, 0.99, 1.0

def discretize(obs):
    return hash(tuple(np.round(obs, 1))) % 10**4

for ep in range(1000):
    obs, _ = env.reset()
    s = discretize(obs)
    total_reward = 0
    while True:
        # ε-greedy: explore or exploit
        a = env.action_space.sample() if np.random.rand() < epsilon else np.argmax(Q[s])
        next_obs, r, done, _, _ = env.step(a)
        s2 = discretize(next_obs)

        # Q-learning update (Bellman equation)
        Q[s, a] += alpha * (r + gamma * np.max(Q[s2]) - Q[s, a])
        s = s2
        total_reward += r
        if done: break

    epsilon = max(0.01, epsilon * 0.995)  # decay exploration
    if ep % 100 == 0:
        print(f"Ep {ep} | reward={total_reward:.0f} | ε={epsilon:.3f}")

python — PPO via Stable-Baselines3 (production pattern)

from stable_baselines3 import PPO, SAC
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.callbacks import EvalCallback

# Vectorised envs for parallel experience collection
train_env = make_vec_env('LunarLander-v2', n_envs=8)
eval_env  = make_vec_env('LunarLander-v2', n_envs=4)

# PPO for discrete action spaces (games, logistics)
ppo = PPO(
    'MlpPolicy', train_env,
    n_steps=2048, batch_size=64, n_epochs=10,
    gamma=0.99, gae_lambda=0.95, clip_range=0.2,
    ent_coef=0.01,   # entropy bonus for exploration
    learning_rate=3e-4, verbose=1
)

# Auto-evaluate + save best model
cb = EvalCallback(eval_env, best_model_save_path='./best', eval_freq=5000)
ppo.learn(total_timesteps=500_000, callback=cb)

# SAC for continuous action spaces (robotics)
# sac = SAC('MlpPolicy', 'Pendulum-v1', verbose=1)
# sac.learn(total_timesteps=100_000)

python — DQN replay buffer + target network (core components)

import torch, torch.nn as nn
from collections import deque
import random

class QNet(nn.Module):
    def __init__(self, obs_dim, act_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, act_dim)
        )
    def forward(self, x): return self.net(x)

replay = deque(maxlen=100_000)     # experience replay buffer
online_net = QNet(obs_dim=4, act_dim=2)
target_net = QNet(obs_dim=4, act_dim=2)
target_net.load_state_dict(online_net.state_dict())  # sync initially
opt = torch.optim.Adam(online_net.parameters(), lr=1e-3)

# Training step
def update():
    batch = random.sample(replay, 64)
    s, a, r, s2, done = zip(*batch)
    s, s2 = torch.FloatTensor(s), torch.FloatTensor(s2)
    a, r, done = torch.LongTensor(a), torch.FloatTensor(r), torch.FloatTensor(done)

    q_vals = online_net(s).gather(1, a.unsqueeze(1)).squeeze()
    with torch.no_grad():
        target = r + 0.99 * target_net(s2).max(1)[0] * (1 - done)

    loss = nn.functional.mse_loss(q_vals, target)
    opt.zero_grad(); loss.backward(); opt.step()

The most frequently asked RL interview questions at DeepMind, OpenAI, Google Brain, and ML engineering roles involving RL or RLHF.

Q1

What is the exploration vs exploitation tradeoff?

Exploitation: always pick the action with the highest known Q-value — maximise immediate reward. Exploration: try new actions — might discover better strategies. Without exploration, agent gets stuck in local optima. Without exploitation, agent never leverages what it learned. ε-greedy balances both: random action with prob ε, best known action otherwise. ε decays over training — explore early, exploit later. Advanced strategies: UCB (upper confidence bound), Thompson sampling, intrinsic motivation (reward for novel states).

ε-greedy: explore prob ε → random · exploit prob (1-ε) → argmax Q · ε: 1.0 → 0.01

Q2

On-policy vs off-policy — what's the difference?

On-policy: learns the value of the policy currently being followed. Must generate experience with the policy being evaluated/improved. Example: SARSA uses the actual next action taken. Can't reuse old data. Off-policy: learns about a target policy while generating experience with a different behaviour policy. Can learn from any data source, including experience replay. Example: Q-learning learns optimal Q* regardless of how actions were selected. Off-policy is more sample-efficient (replay buffer reuse) but can be unstable (DQN's target network addresses this).

On-policy: SARSA, PPO, A2C · Off-policy: Q-learning, DQN, SAC, TD3 · off = reuse data

Q3

Why does DQN use experience replay and a target network?

Experience replay: without it, training samples are highly correlated (consecutive frames from the same episode) — standard SGD assumptions (i.i.d. data) are violated, causing oscillation and divergence. Random sampling from a replay buffer decorrelates the data. Also allows data reuse (each transition can be learned from multiple times). Target network: if you use the same network to compute both the current Q-estimate and the TD target, the target is non-stationary — as the network changes, the target changes too, creating instability. The frozen target network provides stable targets for several thousand updates before being synchronised.

Replay: i.i.d. samples, data reuse · Target net: stable TD targets · both prevent divergence

Q4

What is the discount factor γ and how do you choose it?

γ∈[0,1] controls how much the agent values future rewards relative to immediate ones. γ=0: only immediate reward matters — myopic. γ=1: all future rewards count equally — only for episodic tasks with guaranteed termination. γ=0.99: rewards decay slowly — agent plans far into the future. Choosing γ: high γ (0.99) for tasks where long-term planning matters (chess, Go, robot manipulation). Lower γ (0.9) for tasks where immediate feedback dominates or training instability is a concern. γ also affects the effective "horizon" = 1/(1-γ): γ=0.99 → horizon of 100 steps.

γ=0.99: horizon=100 steps · γ=0.9: horizon=10 steps · higher γ=more future-oriented

Q5

Explain RLHF — why use it for LLMs?

LLMs pretrained on next-token prediction don't inherently know what's helpful or safe — they just predict likely text. RLHF addresses this in three steps: (1) SFT: fine-tune on human-written demonstrations of helpful behaviour. (2) Reward Model: train a model that predicts human preference between two responses. (3) PPO: optimise the LLM to produce responses that score highly on the reward model, with KL penalty to prevent the model from exploiting the reward model (reward hacking). DPO is a simpler alternative that achieves similar results without the explicit reward model and RL phase.

SFT → RM on preference pairs → PPO+KL → aligned LLM · DPO: skip RM+PPO entirely

Q6

What is the credit assignment problem in RL?

In RL, rewards are often delayed — you might not receive a reward until the end of a long episode. The credit assignment problem: which of the many actions taken earlier were responsible for the eventual reward? A chess player who wins can't easily attribute the win to a single move made 30 moves ago. Solutions: discount factor (exponential decay assigns more credit to recent actions), eligibility traces (TD-λ maintains a trace of recently visited states), advantage estimation (GAE in PPO), and attention mechanisms in transformer-based policies.

Delayed reward: which action caused it? · Solutions: γ, eligibility traces, GAE, attention

Q7

When would you choose PPO vs SAC?

PPO: discrete action spaces (games, text generation), on-policy (no replay buffer needed), simpler to implement, stable and well-understood. Default for RLHF, game playing, robotics with moderate data budget. SAC: continuous action spaces (robot joint control, locomotion), off-policy (replay buffer = more sample efficient), maximum entropy exploration, better for tasks requiring smooth continuous control. More complex to implement, may be unstable without careful tuning. Rule: games/discrete → PPO; robot control/continuous → SAC.

Discrete / RLHF / games → PPO · Continuous control / robotics → SAC · both: strong defaults

Everything you need for RL interviews and projects — algorithm selection, key formulas, and the core decision framework.

Algorithm Selection Guide

Situation

Algorithm

Library

Notes

Discrete actions, game/sim

PPO

stable-baselines3

Default for most tasks

Continuous control, robotics

SAC

stable-baselines3

Off-policy, efficient

Continuous, deterministic

TD3

stable-baselines3

Twin critics, stable

LLM alignment

PPO + KL / DPO

TRL (Hugging Face)

DPO simpler now

Game AI from pixels

DQN / Rainbow

stable-baselines3

Replay + target net

Planning with model

MCTS + RL

custom

AlphaZero pattern

Tabular / small state space

Q-learning / SARSA

numpy

No neural net needed

Key Formulas

Discounted Return

Gₜ = Σₖ₌₀^∞ γᵏ rₜ₊ₖ

γ=0.99 typical · horizon = 1/(1-γ) ≈ 100 steps

Q-Learning Update

Q(s,a) ← Q + α[r + γ·max Q(s') - Q]

Off-policy TD · Bellman optimality · α=0.001

Policy Gradient

∇J = E[∇log π(a|s) · A(s,a)]

Increase prob of high-advantage actions

PPO Clip Loss

L = E[min(rA, clip(r,1-ε,1+ε)·A)]

r = π_new/π_old · ε=0.2 · prevents large updates

Advantage Function

A(s,a) = Q(s,a) − V(s)

How much better than average · reduces variance

RLHF Reward

r = RM(response) − β·KL(π||π_SFT)

β prevents reward hacking · β=0.1–0.5 typical

Bellman Optimality

Q*(s,a) = R + γ·Σ P(s')·max Q*(s')

Recursive value decomposition · basis of all RL

SAC Objective

J(π) = E[Σ rₜ + α·H(π(·|sₜ))]

Maximum entropy · α = temperature · encourages exploration

Core Concepts at a Glance

MDP

(S, A, P, R, γ)

State, Action, Transition, Reward, Discount. The formal framework. P unknown in practice → model-free RL.

On vs Off

Policy

On-policy: learn from current policy (PPO). Off-policy: learn from any data, use replay buffer (SAC, DQN). Off = more efficient.

V vs Q

Value Functions

V(s) = value of being in state. Q(s,a) = value of taking action in state. Q more useful — directly tells you what to do.

DPO

vs RLHF

DPO: no reward model, no PPO — optimise preference directly. Simpler, stable, same performance. Now preferred for LLM alignment.

3 rules for RL in production: (1) Always start with a strong non-RL baseline — many RL problems can be solved with supervised imitation learning at a fraction of the sample complexity. (2) Reward shaping is powerful but dangerous — small reward design errors lead to unexpected behaviours (reward hacking). (3) RL is sample-inefficient — expect to need 10–1000× more environment interactions than you'd expect. Use simulation, parallelism, and experience replay.