Case Study Practice
The analyst interview: structuring an open question, choosing the metric, and saying what you would do about it.
What The Case Round Tests
Candidates lose these rounds in predictable ways: jumping straight to a query, computing something impressive that answers a different question, or finishing with an observation instead of a decision. The technical part is rarely what fails.
A Framework That Holds Up
| Step | Do |
|---|---|
| 1. Clarify | What decision does this support, for whom, by when |
| 2. Define | State the metric precisely — a sign-up is what, exactly |
| 3. Structure | Break the question into parts before touching data |
| 4. Hypothesise | List candidate causes, cheapest to check first |
| 5. Analyse | Test them, and say what would falsify each |
| 6. Recommend | What you would do, and what you would watch |
Say the structure out loud before analysing. Interviewers are scoring the approach, and a stated plan lets them redirect you early — which is help, not a penalty.
Choosing The Metric
Half the answer is picking the right number. A metric worth using is specific, comparable over time, and moved by things the team controls.
| Trap | Better |
|---|---|
| Total users | Weekly active users — cumulative counts only go up |
| Average revenue per user | Median, or a distribution — one whale distorts a mean |
| Raw conversion | Conversion by cohort — mix shifts fake movement |
| A vanity total | A rate with a stated denominator |
Diagnosing A Drop
A structured decomposition beats intuition, and it is what the interviewer wants to see.
| Cut by | Reveals |
|---|---|
| Time | A step change (a deploy) versus a slope (a trend) |
| Platform and device | An iOS release, a broken browser build |
| Geography | A regional outage, a campaign that ended |
| Acquisition channel | Paid spend stopping, an SEO change |
| Funnel step | Which stage lost the users |
| New versus returning | Acquisition problem or retention problem |
Always ask whether the data itself broke first: a tracking change, a failed pipeline run or a bot filter deployed last Tuesday explain a surprising share of "drops", and finding that is a good answer, not a cop-out.
Ending With A Recommendation
Lead with the answer. Stakeholders need the conclusion, its confidence, and what to do — the methodology is support, not the story.
| Say | Not |
|---|---|
| "Sign-ups fell 20%, all of it on Android after Tuesday's release" | "I joined the events table to the users table…" |
| "I'd roll back and confirm within a day" | "Here is a dashboard" |
| "Confident on the where, less on the why" | Unqualified certainty |
| "If I had more time I'd check X" | Silence about the gaps |
Interview Questions
How do you approach an open case question?
Clarify the decision it supports, define the metric precisely, structure the question into parts, list hypotheses cheapest-first, test them, and finish with a recommendation and what you would monitor.
Sign-ups dropped 20% — what do you check first?
Whether the data broke: tracking changes, a failed pipeline, a bot filter. Then a deploy, a campaign ending, or a platform-specific release — before theorising about user behaviour.
What makes a metric worth using?
It is specific, comparable over time, and moved by things the team controls. Cumulative totals and averages distorted by outliers fail all three.
What is Simpson's paradox in this context?
Every segment improves while the overall number worsens, because the mix between segments shifted. It is why you segment before concluding a trend is real.
How should a case answer end?
With a recommendation and a stated confidence level — what you would do, what you would watch to confirm it, and what you would check with more time.
Why state uncertainty explicitly?
It reads as senior. Separating what the data shows clearly from what remains ambiguous is more useful, and more credible, than unqualified confidence.