Monitoring & Alerts
Metrics, logs, traces, SLIs/SLOs, on-call hygiene.
Theory
Monitoring is the practice of collecting, storing, and alerting on metrics that describe system behavior. The four golden signals (Google SRE): Latency (how long requests take), Traffic (requests per second), Errors (rate of failed requests), Saturation (how full the system is — CPU/memory/queue depth). Every service should expose and monitor these four.
SLOs, SLIs, and SLAs: SLI (Service Level Indicator) is a measured metric — p99 latency. SLO (Service Level Objective) is the target — p99 < 200ms for 99.9% of requests. SLA (Service Level Agreement) is a contractual commitment with penalties. Internal teams set SLOs; customer-facing commitments are SLAs. Error budgets = 1 - SLO — the allowed failure margin in a given period.
The monitoring stack: metrics (Prometheus scrapes, stores, and alerts), visualization (Grafana dashboards), log aggregation (Loki, Elasticsearch, CloudWatch Logs), distributed tracing (Jaeger, Tempo, Zipkin), alerting (Alertmanager, PagerDuty). OpenTelemetry is the vendor-neutral standard for instrumenting apps to emit metrics, logs, and traces in a consistent format.
Alert design: alert on symptoms (user-visible pain — high error rate, high latency) not causes (CPU 90%). A CPU alert at 9am for a batch job is noise; an error rate alert is a page. Alert fatigue occurs when too many low-signal alerts train on-call to ignore pages. Each alert must have a runbook: what it means, what to check, how to fix it, and who owns it.
On-call rotations: PagerDuty, OpsGenie, or VictorOps route alerts to the on-call engineer. Escalation policies: alert the primary, if no ack in 10 minutes escalate to secondary, then to the manager. On-call compensation and sustainable schedules (no more than 1 week/month per engineer, no repeated nights) are SRE best practices. Post-incident reviews improve systems and reduce future pages.
Observability vs monitoring: monitoring tells you what is broken (alert: error rate > 5%). Observability tells you why — you can ask arbitrary questions about system state by exploring traces, logs, and metrics together without pre-defining every possible failure mode. Well-instrumented systems emit structured logs with trace IDs, enabling correlation across the three pillars: metrics → traces → logs.
Synthetic monitoring: Blackbox Exporter (Prometheus), Pingdom, or Datadog Synthetics actively probe endpoints (HTTP GET, TCP check, SSL cert expiry) from external locations, independent of your internal monitoring stack. Catches CDN failures, DNS outages, and certificate expiry that internal metrics miss because the issue is between your LB and the user.
Architecture Diagram
Users / clients
|
Monitoring & Alerts
|
Core services
|
Data + observabilityExamples
# Monitoring & Alerts
# Metrics, logs, traces, SLIs/SLOs, on-call hygiene.
# Validate in staging before production rollout.Interview Questions
What problem does Monitoring & Alerts solve?
It addresses the core use case described in production architecture — map features to reliability, scale, or velocity outcomes.
Key components of Monitoring & Alerts?
Identify inputs, outputs, control plane, data plane, and failure domains — interviewers want structured decomposition.
Common production pitfalls?
Misconfiguration, missing observability, no rollback path, and scaling bottlenecks under peak load.
How do you test changes safely?
Staging parity, canary/gradual rollout, automated health checks, and documented rollback.
Metrics to prove success?
Error rate, latency percentiles, throughput, cost, and toil reduction — pick one primary SLO.
Beginner vs advanced concern?
Beginners focus on setup; advanced teams focus on blast radius, security boundaries, and operability at 10× scale.
Best Practices
- Treat Monitoring & Alerts config as code with review and CI validation.
- Define SLOs and dashboards before production cutover.
- Document rollback and ownership for on-call.
- Use least privilege for credentials.
Common Mistakes
- Adopting Monitoring & Alerts without measurable success criteria.
- No staging environment mirroring production constraints.
- Missing rollback path during incidents.
- Undocumented on-call expectations.
Cheat Sheet
Practical Exercises
Stand up Monitoring & Alerts locally or in free tier; document commands and failure recovery.
Introduce misconfiguration; practice detection and rollback under time limit.