SLOs & Reliability
Service level objectives and error budgets, and the on-call, toil and postmortem practices that turn reliability into a decision rather than a hope.
Reliability Is a Number, Not a Feeling
Without an agreed number, every incident becomes an argument and every release a standoff between people who want to ship and people who want it to stay up. An objective ends that argument by turning it into arithmetic.
Three terms, often confused. An SLI is a measurement. An SLO is your internal target for it. An SLA is a contract with a customer, with money attached — and it should always be looser than your SLO, so you find out you are in trouble before they do.
| Term | Is | Example |
|---|---|---|
| SLI | What you measure | Share of requests served under 300ms |
| SLO | Your target for it | 99.9% of them, over 30 days |
| SLA | The contractual promise | 99.5%, or credits are owed |
Choosing Good SLIs
A good indicator measures what the user experiences. CPU utilisation is a fine dashboard metric and a terrible SLI — nobody has ever noticed your CPU. They notice slow pages and errors.
| Kind | Measures | Typical shape |
|---|---|---|
| Availability | Did the request succeed | Non-5xx responses ÷ all responses |
| Latency | Was it fast enough | Share served under a threshold |
| Quality | Was the answer complete | Full results vs degraded fallback |
| Freshness | How stale is the data | Age of the newest record |
Keep the set small. Two or three indicators per service that genuinely track user pain beat twenty that nobody reads. Every SLI you add is one more thing to argue about during an incident.
Spending an Error Budget
The error budget is what the objective leaves you. A 99.9% SLO over 30 days permits about 43 minutes of failure — and that is a budget to spend, not a disaster to avoid.
| SLO | Budget per 30 days |
|---|---|
| 99% | about 7 hours |
| 99.9% | about 43 minutes |
| 99.95% | about 22 minutes |
| 99.99% | about 4 minutes |
The budget is what makes the model useful, because it converts reliability into a shared decision. Budget remaining: ship, take risks, run experiments. Budget exhausted: releases stop and reliability work takes priority until it recovers.
Consistently unspent budget is also information: the objective is too loose, or you are being too cautious and could be shipping faster. Reliability far beyond the target costs real money and buys nothing a user notices.
Toil & Automation
Toil is work that is manual, repetitive, automatable, and grows with the size of the service while adding no lasting value. Restarting a stuck process by hand every week is toil. Writing the thing that restarts it is not.
This is the distinction between an SRE and a sysadmin: not the tasks, but whether you absorb them or eliminate them. Absorbing toil scales linearly with traffic, and eventually there is no time left for anything else.
| Toil | Not toil |
|---|---|
| Manually restarting a service | Writing the health check that restarts it |
| Copying config between environments | Putting the config in version control |
| Answering the same question weekly | Writing the runbook once |
| Applying the same fix repeatedly | Fixing the underlying cause |
On-Call Practice
On-call is sustainable or it is not, and the difference is mostly page quality. A rotation that pages for things nobody acts on trains people to ignore it, which is worse than not paging at all.
| Practice | Why |
|---|---|
| Page only on user-visible impact | Everything else is a ticket, not a 3am call |
| Every page has a runbook | The responder is tired and may not own the service |
| Enough people in the rotation | One person on call permanently is attrition with a schedule |
| Handover between shifts | Ongoing problems should not be rediscovered |
| Time back after a bad night | Otherwise the cost is invisible and never fixed |
During an incident, separate the roles: someone commands, someone communicates, someone investigates. When one person does all three, the communication is what gets dropped — and the absence of updates is what turns an outage into a crisis.
Blameless Postmortems
The point is to change the system so the same failure cannot recur. "Blameless" is not politeness — it is the only way to get accurate information, because people who expect blame report less.
"Human error" is never a root cause. If one mistyped command took production down, the finding is that a single command could take production down. The person is the last link in a chain, not the explanation.
| Section | Contains |
|---|---|
| Impact | Who was affected, how badly, for how long |
| Timeline | What happened and when, in facts |
| Detection | How you found out — and how long that took |
| Root cause | The conditions that made it possible |
| Actions | Specific, owned, dated |
| What went well | Preserve what worked |
A postmortem with no owned, dated actions is a document. The follow-through is the entire value, and unactioned postmortems are why the same incident happens twice.
Capacity Planning
Knowing when you will run out, before you do. Autoscaling handles the daily curve; it does not save you from a quota ceiling, a database connection limit, or a launch nobody told you about.
| Watch | Because |
|---|---|
| Growth trend per resource | Tells you the date, not just the number |
| Hard limits and quotas | Connections and API quotas do not autoscale |
| Headroom at peak | Average utilisation hides the peak entirely |
| Load test results | The only honest answer to "can we take 3×" |
| Cost per unit of traffic | Capacity you cannot afford is not capacity |
Interview Questions
SLI, SLO, SLA?
The SLI is the measurement, the SLO is your internal target for it, the SLA is the contractual promise with consequences. Keep the SLA looser than the SLO so you breach your own target first and have time to react.
Why not aim for 100% availability?
It is unattainable and the pursuit stops all change. Beyond a point extra nines cost a great deal and are invisible to users — often below the reliability of their own network. The budget exists to be spent on shipping.
What is an error budget and how is it used?
The unreliability the SLO permits — about 43 minutes a month at 99.9%. Budget remaining means ship and take risks; budget exhausted means releases pause for reliability work. The policy must be agreed before it is needed.
What makes a bad SLI?
One that measures the system rather than the experience. CPU and memory are diagnostics. Users notice errors, slowness and stale data — measure those, as a proportion under a threshold rather than an average.
What is toil, and why cap it?
Manual, repetitive, automatable work that scales with the service and leaves nothing behind. Uncapped, it consumes the time that would have prevented it, so the system degrades while everyone is busy.
Why insist postmortems are blameless?
To get accurate information. People who expect blame report less, and the detail you lose is the detail you needed. "Human error" is not a cause — if one command could break production, that is the finding.