Prometheus

Theory

Prometheus is a pull-based monitoring system: it scrapes HTTP /metrics endpoints on a configured interval (default 15s). Each target exposes metrics in the Prometheus text format — one metric per line with optional labels in {curly braces}.

Four metric types: Counter (monotonically increasing, e.g. http_requests_total), Gauge (can go up or down, e.g. memory_usage_bytes), Histogram (bucketed observations for latency, e.g. http_request_duration_seconds_bucket), Summary (precomputed quantiles on client side).

PromQL is Prometheus's query language. rate(http_requests_total[5m]) computes per-second rate over a 5-minute window. irate() uses only the last two samples — more reactive but noisier. histogram_quantile(0.99, ...) computes the p99 latency from histogram buckets.

Recording rules pre-aggregate expensive queries and store results as new time series. Alert rules evaluate PromQL expressions and fire to Alertmanager when the condition is true for a specified duration (for: 5m prevents flapping alerts).

Alertmanager handles routing, grouping, silencing, and inhibition of alerts. Routes match alert labels (e.g. severity=critical → PagerDuty, severity=warning → Slack). Group_by merges related alerts into one notification to reduce noise.

Exporters translate third-party metrics into Prometheus format. node_exporter exposes host OS metrics (CPU, memory, disk, network). blackbox_exporter probes HTTP/TCP endpoints for availability. Custom apps should expose /metrics natively using client libraries (Python, Go, Java).

Service discovery via kubernetes_sd_config, ec2_sd_config, or static_configs. Relabeling (relabel_configs) filters or rewrites target labels before scraping — drop dev namespaces, rename label keys, add env labels from instance metadata.

Prometheus stores data locally in a TSDB with 2-hour blocks compacted into larger chunks. Default retention is 15 days. For long-term storage, remote_write to Thanos, Cortex, or Mimir — these also enable multi-cluster federation and HA.

Architecture Diagram

Exporters / app /metrics
         | scrape (pull)
  Prometheus TSDB
         |
    +----+----+
    v         v
  Grafana    Alertmanager
  (viz)      (route/notify)

Examples

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alerts.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: "app"
    static_configs:
      - targets: ["app:8000"]
    metrics_path: /metrics

  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"

groups:
  - name: app_alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) /
              rate(http_requests_total[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for {{ $labels.job }}"

      - record: job:http_request_duration_seconds:p99
        expr: histogram_quantile(0.99, sum(rate(
              http_request_duration_seconds_bucket[5m])) by (le, job))

# Request rate per endpoint
rate(http_requests_total[5m])

# p99 latency
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, handler))

# CPU usage %
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory available %
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100

# 5xx error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) /
sum(rate(http_requests_total[5m]))

Interview Questions

Best Practices

• Use Counters for requests/errors/bytes — never a Gauge. Use rate() on Counters, never on Gauges.
• Choose histogram bucket boundaries that bracket your expected p50/p90/p99 — default buckets rarely fit your latency distribution.
• Add for: 5m to every alert rule to prevent flapping alerts on momentary spikes.
• Alert on symptoms (high error rate, high latency) not causes (CPU 80%) — fewer, more actionable pages.
• Use recording rules to pre-aggregate expensive PromQL — dashboards load in milliseconds instead of scanning millions of series.
• Never put high-cardinality values (user_id, request_id) in label sets — one label value = one time series = potential OOM.

Common Mistakes

• Using rate() on a Gauge — Gauges go up and down, rate() gives nonsensical results. Use avg() or max() on Gauges directly.
• Setting scrape_interval too short (e.g. 1s) — creates storage pressure and makes irate() meaningless. 15–30s is standard.
• Alerting on CPU/memory thresholds without context — 90% CPU is fine under expected load; alert on SLO burn instead.
• Using client-side Summary for aggregated quantiles — quantiles cannot be re-aggregated across instances. Use Histogram instead.
• No up == 0 alert — when a scrape target disappears, Prometheus records no data (no alert fires). You need an explicit absent/up check.

Cheat Sheet

Practical Exercises