Optimisers & Training Tricks
Gradient descent and its descendants, learning-rate schedules, and reading a loss curve.
Descending A Surface You Cannot See
| Variant | Gradient from | Character |
|---|---|---|
| Batch | The whole dataset | Accurate, slow, memory-hungry |
| Stochastic | One example | Noisy, fast, escapes shallow minima |
| Mini-batch | 32–512 examples | The practical compromise, and what everyone uses |
The noise in mini-batch gradients is not purely a defect. It helps the optimiser escape poor regions, which is part of why very large batches do not automatically train better models.
SGD, Momentum & Adam
| Optimiser | Idea | Reality |
|---|---|---|
| SGD | Step against the gradient | Slow through ravines, sensitive to the rate |
| SGD + momentum | Accumulate a velocity | Smoother, faster; still the best final accuracy in vision |
| RMSProp | Per-parameter step from recent gradient size | Handles wildly different scales |
| Adam | Momentum plus RMSProp | Converges fast with little tuning — the default |
| AdamW | Adam with weight decay decoupled | What transformers are actually trained with |
The practical advice is unglamorous: start with AdamW and a sensible learning rate. It converges quickly with almost no tuning. If you are chasing the last point of accuracy on a vision benchmark, tuned SGD with momentum still tends to generalise slightly better.
Learning Rate & Batch Size
The learning rate is the hyperparameter that matters most. Too high and the loss oscillates or diverges; too low and training crawls into whatever minimum is nearest.
| Technique | Why | |
|---|---|---|
| Warmup | Large early steps on random weights destabilise training | |
| Cosine or step decay | Big steps to explore, small steps to settle | |
| Learning-rate finder | Sweep the rate and watch where loss starts falling | |
| Scale rate with batch size | A larger batch gives a less noisy gradient, so it tolerates a bigger step |
opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=3e-4, total_steps=steps)
for batch in loader:
loss = criterion(model(batch.x), batch.y)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) # tame exploding gradients
opt.step(); sched.step(); opt.zero_grad(set_to_none=True)Gradient clipping bounds the update size and is close to mandatory for recurrent networks and transformers, where a single bad batch can otherwise blow the weights up irrecoverably. Mixed precision — computing in 16-bit with a 32-bit master copy — roughly halves memory and speeds training up substantially on modern hardware.
Reading The Loss Curve
| Curve | Means | Do |
|---|---|---|
| Loss flat from the start | Rate too low, or a bug — check the data pipeline | Raise the rate; overfit one batch as a sanity test |
| Loss oscillating or NaN | Rate too high, or exploding gradients | Lower the rate, clip gradients, check for divide-by-zero |
| Training falls, validation rises | Overfitting from this epoch on | Early stop, regularise, get more data |
| Both plateau early and high | Underfitting | More capacity, better features, train longer |
| Sudden spike mid-training | A bad batch or a data corruption | Clip, shuffle, inspect that batch |
Interview Questions
Why mini-batch rather than full-batch gradient descent?
Full batches are accurate but slow and memory-bound. Mini-batches update far more often, and their gradient noise helps escape poor regions of the loss surface.
SGD with momentum or Adam?
Adam, or AdamW, converges fast with little tuning and is the sensible default. Tuned SGD with momentum still tends to generalise slightly better on vision benchmarks.
Why use learning-rate warmup?
Early in training the weights are random and gradients are large, so full-size steps destabilise the run. Warmup ramps the rate up over the first few hundred steps.
What does gradient clipping prevent?
A single batch producing an enormous update that destroys the weights. It is close to mandatory for RNNs and transformers.
What does a loss that will not move indicate?
Usually a learning rate far too low or a bug in the data pipeline. Overfitting a single batch distinguishes them: if that fails, the problem is not the optimiser.
Why does Adam use more memory than SGD?
It stores two moment estimates per parameter, so optimiser state is roughly twice the model size — a real constraint on large models.