Mathematics for ML
Linear algebra, calculus and probability — the working subset you need to read a paper and debug a model that will not converge.
How Much Do You Actually Need
You will almost never derive anything by hand. Libraries do the arithmetic. What the mathematics buys you is the ability to reason about why something behaves the way it does — why the loss plateaued, why the features had to be scaled, why that layer's gradients vanished.
Three areas carry nearly all the weight. Linear algebra is how data and models are represented. Calculus is how they learn. Probability is how you deal with the fact that none of it is certain.
Learn them alongside the models rather than as a prerequisite course. Abstract linear algebra is dull; understanding that a neural network layer is a matrix multiply followed by a nonlinearity makes it stick.
Linear Algebra
Everything in machine learning is an array of numbers. A sample is a vector, a batch is a matrix, an image batch is a tensor. Almost every operation is a matrix multiplication.
| Object | Shape | In practice |
|---|---|---|
| Scalar | — | A learning rate, a loss value |
| Vector | n | One sample's features; one embedding |
| Matrix | m × n | A batch of samples; a layer's weights |
| Tensor | 3D+ | Images (batch, height, width, channels) |
The dot product is the single most important operation, because it measures alignment. Two vectors pointing the same way give a large positive number; perpendicular gives zero. That is exactly what cosine similarity uses, which is why it underpins search, recommendations and retrieval-augmented generation.
# a batch of 32 samples, 10 features each X = shape(32, 10) # a layer mapping 10 features to 64 units W = shape(10, 64) # inner dimensions must match: 10 == 10 XW = shape(32, 64) # the error you will see a thousand times: # mat1 and mat2 shapes cannot be multiplied (32x10 and 64x10)
Matrix multiplication is not commutative: AB and BA are different, and often only one is even defined. Nearly every shape error is the inner dimensions failing to meet.
Two more worth knowing by name: the transpose, which flips rows and columns and is how you make dimensions line up; and eigenvectors, the directions a transformation only stretches, which is what PCA finds when it picks the axes carrying the most variance.
Calculus & Gradients
Calculus is how a model learns. A derivative answers one question: if I nudge this parameter, which way does the error move, and how sharply?
With many parameters the derivative becomes a gradient — a vector holding one partial derivative per parameter. It points in the direction of steepest increase, so training walks the opposite way. That is the whole of gradient descent: compute the gradient, step against it, repeat.
for epoch in range(n): preds = model(X) loss = error(preds, y) grad = d_loss_d_weights(loss) # which way is uphill weights -= learning_rate * grad # step downhill
The learning rate is the step size, and it explains most training failures. Too large and the loss oscillates or diverges; too small and it crawls or settles in a poor spot. It is the first thing to change when training misbehaves.
The chain rule is what makes deep networks trainable. To know how a weight in the first layer affects the final loss, you multiply the derivatives along the path between them. That is all backpropagation is: the chain rule, applied backwards through the network, reusing the intermediate results.
Probability
Machine learning outputs beliefs, not facts. A classifier does not say "spam" — it says 0.94, and what you do at 0.94 is a decision you have to make.
| Idea | Why it matters here |
|---|---|
| Distribution | The shape of the data decides which model and metric are sensible |
| Conditional probability | P(spam given these words) is literally what a classifier estimates |
| Bayes' theorem | Updating a belief with evidence; the base rate usually dominates |
| Expectation | The long-run average — what a loss function is minimising |
| Variance | How much predictions move if you resample the training data |
| Independence | Assumed constantly, true rarely; Naive Bayes is named after it |
The normal distribution shows up everywhere because of the central limit theorem: averages of many independent things tend toward it regardless of what they started as. That is why so many methods assume it — and why they mislead on the heavy-tailed data that is common in practice.
Cross-entropy, the standard classification loss, is a probability idea: it measures how surprised the model is by the true answer. Confident and right costs almost nothing; confident and wrong costs a great deal. That asymmetry is what pushes a model toward honest probabilities rather than confident guesses.
Interview Questions
What is a gradient, and what does gradient descent do with it?
The gradient is the vector of partial derivatives of the loss with respect to each parameter; it points uphill. Gradient descent steps in the opposite direction by a distance set by the learning rate, and repeats.
Why do gradients vanish in deep networks?
Backpropagation multiplies derivatives along the path from loss to weight. Many factors below one drive the product toward zero, so early layers receive almost no signal. ReLU, residual connections and normalisation are all responses to this.
Why does cosine similarity dominate embedding search?
It compares direction rather than magnitude, so two texts about the same subject score as similar even when one is far longer. It is the dot product of the normalised vectors.
Explain the bias–variance trade-off.
Bias is error from a model too simple to represent the pattern; variance is error from sensitivity to the particular training sample. Reducing one usually raises the other, and the aim is the total, not either alone.
Why is the normal distribution assumed so often?
The central limit theorem: sums and averages of many independent contributions tend toward it. It is convenient and often reasonable — and badly wrong on heavy-tailed data like income or latency.
Your loss is oscillating and will not come down. First thing you change?
The learning rate — lower it. Oscillation is the classic sign of steps too large to settle into a minimum.