Observability
Centralised logging, distributed tracing and alerting — the three things that turn “it is slow” into a cause.
Monitoring Answers Known Questions
Three signals, and they do different jobs. Metrics are cheap numbers over time — good for dashboards and alerts, bad at explaining a single request. Logs are detailed events — good for explaining, expensive at volume. Traces follow one request across services — the only one that answers "where did the time go".
| Signal | Answers | Cost |
|---|---|---|
| Metrics | Is it happening, how often, how bad | Cheap, aggregated |
| Logs | What exactly happened here | Expensive at volume |
| Traces | Where the latency went across services | Sampled, needs instrumenting |
The usual path is metrics to notice, traces to locate, logs to explain. A team with only logs will find things eventually and expensively; one with only metrics will know something is wrong and not why.
Centralised Logging
Logs on a machine are useless the moment you have more than one machine, or a container that has already exited. Ship them somewhere central and searchable.
Log structured, as key-value or JSON, not prose. A line reading "user 42 failed login from 1.2.3.4" cannot be aggregated; the same event with fields can be counted, grouped and alerted on.
# prose: greppable at best, never aggregatable "Failed login for user 42 from 1.2.3.4 after 3 tries" # structured: countable, groupable, alertable { "event": "login_failed", "user_id": 42, "source_ip": "1.2.3.4", "attempts": 3, "trace_id": "a1b2c3" } # ties this line to the trace
| Rule | Why |
|---|---|
| One event per line | Multi-line entries break every parser |
| Include a trace id | Joins the log to the request it belongs to |
| Use levels honestly | If everything is ERROR, nothing is |
| Never log secrets or personal data | Logs are widely readable and retained |
| Set retention deliberately | Storage is the entire cost of logging |
Distributed Tracing
One user request crossing six services produces six sets of logs with no obvious relationship. A trace stitches them into one timeline.
A trace is the whole request; a span is one unit of work within it, with a duration and a parent. Nested spans reveal where the time actually went — usually somewhere nobody suspected.
| Concept | Is |
|---|---|
| Trace id | Follows the request everywhere it goes |
| Span | One operation, timed, with a parent |
| Context propagation | Passing the ids across service boundaries |
| Sampling | Keeping a fraction, because all of it is too much |
Sample intelligently: a small share of successful requests is plenty, but keep everything that errored or ran slowly. Those are the traces you will actually open.
Alerting
An alert is a claim that a human must act now. If that is not true, it is not an alert — it is a dashboard, or a ticket.
Alert on symptoms, not causes. "Error rate above 2% for five minutes" matters whatever caused it. "CPU above 80%" may be perfectly healthy, and firing on it teaches people that pages are usually noise.
| Page | Do not page |
|---|---|
| Users are seeing errors | A single pod restarted |
| Latency breached the objective | CPU is briefly high |
| Error budget burning fast | Disk at 60% |
| A queue is growing without bound | A deploy finished |
Use burn rate where you have an SLO: alert when the error budget is being consumed fast enough to run out early. It fires for real problems and stays quiet for brief blips, which fixed thresholds cannot do.
Interview Questions
Monitoring versus observability?
Monitoring answers questions you defined in advance — dashboards and alerts for known failure modes. Observability is whether you can answer a new question from existing data, without shipping code. Metrics monitor; traces and structured logs give observability.
Why structured logs?
Prose can only be grepped. Structured events can be counted, grouped, filtered and alerted on, and can carry a trace id linking them to the request they came from.
What is a span?
One timed operation within a trace, with a parent. Nesting them shows where the latency went across services — which no single service's logs can tell you.
Symptoms or causes — what do you alert on?
Symptoms. Users care about errors and slowness, not CPU. Cause-based alerts fire when nothing is wrong and train people to ignore the pager.
What is burn-rate alerting?
Alerting on how fast the error budget is being consumed rather than on a fixed threshold. Fast burn pages immediately; slow burn raises a ticket. It ignores brief blips that no fixed threshold can distinguish.
Why is tracing often incomplete?
Context propagation. Every service must forward the trace headers; one that does not silently truncates the trace, and everything downstream disappears with no error.