Chaos Engineering
Breaking things deliberately, in daylight, with a hypothesis — because they will break anyway.
Failure Is Already Scheduled
The premise is that a system of any size is continuously partly broken: an instance dies, a zone degrades, a dependency slows down. Those events are certain. The only choice is whether you first meet them at 3am with a customer on the phone, or at 2pm with the team assembled.
What it usually finds is not that a component failed — that was the point — but that the response to failure was wrong: a retry storm, a missing timeout, an alert that never fired, a runbook naming someone who left last year.
Designing An Experiment
| Step | Detail |
|---|---|
| 1. Define steady state | A measurable normal — checkout success rate, not CPU |
| 2. State a hypothesis | "Losing one replica will not change checkout success" |
| 3. Bound the blast radius | One instance, one zone, 1% of traffic |
| 4. Know how to stop | An abort condition and a way to undo it instantly |
| 5. Inject and observe | Watch the steady-state metric, not the injected thing |
| 6. Fix, then widen | Repeat larger only once the finding is closed |
Start in staging to shake out the obvious, but understand its limit: staging has different data volumes, different traffic and different scale, so it cannot tell you how production behaves. The valuable experiments are the small, well-bounded ones in production.
What To Inject
| Injection | Question it answers |
|---|---|
| Kill an instance or pod | Does traffic move without errors |
| Add latency to a dependency | Do timeouts fire, or do threads pile up |
| Return errors from a dependency | Does the circuit breaker open, is there a fallback |
| Exhaust CPU, memory or disk | Does it degrade or fall over |
| Sever a zone | Does failover work, and is capacity sufficient |
| Expire a certificate | The most common self-inflicted outage there is |
Game Days & Culture
A game day is a scheduled exercise where a team injects a failure and works the response as if it were real. It tests the parts an automated experiment cannot: whether the alert reached a human, whether the runbook was correct, whether anyone knew who was in charge.
| Reveals | Typically |
|---|---|
| Alerting gaps | The failure produced no page at all |
| Stale runbooks | Commands referencing systems that no longer exist |
| Access problems | Nobody on call could reach the dashboard |
| Coordination | No clear incident lead, three people editing the same thing |
| Recovery time | Far longer than anyone estimated |
The output is a list of fixes with owners, and the discipline is closing them before running the same experiment larger. Chaos engineering that generates findings nobody actions is theatre with extra risk.
Interview Questions
What is chaos engineering actually testing?
Not whether a component can fail — that is assumed — but whether the system's response is correct: timeouts, fallbacks, failover, alerting and the humans on call.
What must an experiment have?
A measurable steady state, a stated hypothesis, a bounded blast radius, and an abort condition with a way to undo it instantly.
Why inject latency rather than only failure?
Systems usually cope with a dependency that is down. A dependency that responds slowly exhausts connection and thread pools and takes the caller with it.
Why run experiments in production?
Staging has different data volumes, traffic and scale, so it cannot tell you how production behaves. Production experiments are made safe by bounding them, not by avoiding them.
What does a game day add over automated chaos?
It tests the human path: whether the alert paged someone, whether the runbook was accurate, whether anyone was clearly in charge, and how long recovery actually took.
What makes chaos engineering fail as a practice?
Findings that nobody closes, and a culture where a failed experiment becomes a blame conversation — after which nobody proposes another one.