← DevOps Hub
DevOps · Guide

Grafana

Dashboards, panels, alerts, observability visualization.

— min read DevOps

Theory

Grafana: Dashboards, panels, alerts, observability visualization.

Grafana is an open-source analytics and monitoring platform that visualizes time-series data from multiple sources. It connects to Prometheus, Elasticsearch, Loki (logs), Tempo (traces), InfluxDB, CloudWatch, and 50+ other data sources via plugins. Grafana provides dashboards, alerting, on-call scheduling, and incident management in one UI.

Dashboards are the core unit. A dashboard contains panels — time series charts, stat panels (single value), gauge charts, heatmaps, tables, logs panels. Each panel has a query (PromQL, LogQL, SQL depending on the source), visualization type, and display options. Dashboards are JSON under the hood and should be version-controlled.

Variables make dashboards dynamic. A $job variable (query: label_values(up, job)) populates a dropdown with all Prometheus job labels. Panels reference $job in queries: rate(http_requests_total{job="$job"}[5m]). Users select a job from the dropdown, and all panels update. This turns one dashboard into a multi-service template.

Alerting: define alert rules in Grafana (or in Prometheus) with PromQL/LogQL conditions. Contact points configure where alerts go (Slack, PagerDuty, email, OpsGenie). Notification policies route alerts to contact points based on label matchers. Silence rules suppress alerts during maintenance windows. Mute timings suppress alerts on a recurring schedule (weekends, off-hours).

The RED method for service dashboards: Rate (requests per second), Errors (error rate %), Duration (latency p50/p95/p99). The USE method for infrastructure: Utilization (CPU/memory/disk %), Saturation (queue depth, context switches), Errors. These two patterns cover 90% of what you need to monitor in any system.

Provisioning: Grafana datasources and dashboards can be provisioned via YAML config files (grafana/provisioning/datasources/, grafana/provisioning/dashboards/). This makes Grafana configuration reproducible — spin up a new Grafana instance and all dashboards and datasources are restored automatically. Essential for Kubernetes deployments via Helm chart.

Loki integration: Grafana's log aggregation system. Loki indexes only metadata labels (not log content), storing compressed log chunks in object storage. LogQL is the query language: {app="nginx"} |= "ERROR" | json | status >= 500. Explore view correlates logs with metrics — click a spike on a Prometheus graph, see the log lines at that timestamp.

Architecture Diagram

Users / clients
         |
  Grafana
         |
  Core services
         |
  Data + observability

Examples

bash 
# Grafana
# Dashboards, panels, alerts, observability visualization.
# Validate in staging before production rollout.

Interview Questions

What problem does Grafana solve?

It addresses the core use case described in production architecture — map features to reliability, scale, or velocity outcomes.

Key components of Grafana?

Identify inputs, outputs, control plane, data plane, and failure domains — interviewers want structured decomposition.

Common production pitfalls?

Misconfiguration, missing observability, no rollback path, and scaling bottlenecks under peak load.

How do you test changes safely?

Staging parity, canary/gradual rollout, automated health checks, and documented rollback.

Metrics to prove success?

Error rate, latency percentiles, throughput, cost, and toil reduction — pick one primary SLO.

Beginner vs advanced concern?

Beginners focus on setup; advanced teams focus on blast radius, security boundaries, and operability at 10× scale.

Best Practices

  • Treat Grafana config as code with review and CI validation.
  • Define SLOs and dashboards before production cutover.
  • Document rollback and ownership for on-call.
  • Use least privilege for credentials.

Common Mistakes

  • Adopting Grafana without measurable success criteria.
  • No staging environment mirroring production constraints.
  • Missing rollback path during incidents.
  • Undocumented on-call expectations.

Cheat Sheet

GrafanaDashboards, panels, alerts, observability visualization.
SLOService level objective
RollbackRevert to last known good
CanaryLimited blast-radius rollout
RunbookIncident steps

Practical Exercises

Grafana sandbox

Stand up Grafana locally or in free tier; document commands and failure recovery.

Failure drill

Introduce misconfiguration; practice detection and rollback under time limit.

← Prometheus

Related topics