Experimentation
A/B tests, statistical power, and the peeking problem that makes half of all reported wins imaginary.
Why Experiments Beat Analysis
Randomly assigning users to a control and a variant makes the two groups equivalent on average in every respect, including the ones you never thought to measure. That is the whole trick, and no amount of regression on observational data reproduces it.
For most product teams, running experiments well is worth more than another model. It is also where the most confident wrong conclusions in the industry are produced.
Experiment Design
Decide everything before you look at any data: the metric, the effect size worth caring about, the sample size, and how long it runs. Deciding afterwards is how noise gets shipped as a feature.
| Decide up front | Why it must be up front |
|---|---|
| One primary metric | Twenty metrics guarantee one looks significant |
| Minimum detectable effect | It sets your sample size, and rules out chasing noise |
| Sample size & duration | Stopping when you like invalidates the maths |
| Guardrail metrics | A conversion win that wrecks retention is not a win |
| Unit of randomisation | Usually the user, not the session or the page view |
Power is the chance of detecting an effect that is genuinely there. Underpowered tests are the quiet disaster of the field: they mostly produce null results you cannot interpret, and the wins they do report are exaggerated.
Run for whole weeks. Tuesday behaviour is not Saturday behaviour, and a test that stops mid-week has a weekday bias baked into it.
Common Mistakes
| Mistake | What it produces |
|---|---|
| Stopping as soon as it is significant | False positive rate several times higher than advertised |
| Choosing the metric after seeing results | A story fitted to noise |
| Randomising by session | One user in both arms; the comparison is meaningless |
| Ignoring novelty effects | A win that decays once the change stops being new |
| No guardrails | Shipping a conversion lift that raises churn |
| Segmenting until something wins | Multiple comparisons wearing a disguise |
Interview Questions
Why can you not stop a test the moment it turns significant?
Because significance is defined for a single test at a predetermined sample size. Checking repeatedly and stopping on the first win gives noise many chances to cross the threshold, pushing the real false positive rate far above 5%.
What is statistical power, and why does it matter commercially?
The probability of detecting a real effect. An underpowered test mostly returns inconclusive results, so you spend engineering time on experiments that could never have answered the question.
Your A/B test shows +2% conversion but −5% retention. Ship it?
No. Retention is the guardrail and it moved further than the win. This is exactly why guardrail metrics are chosen before the test rather than after.
Why randomise by user rather than by session?
A user with several sessions would land in both arms, contaminating the comparison and producing an inconsistent experience for them.
The variant wins in every segment but loses overall. How?
Simpson's paradox — the segments have different sizes or base rates. Check the assignment for imbalance before believing either number.