Load Balancing

Theory

Load balancers distribute incoming traffic across multiple backend servers to improve availability, throughput, and fault tolerance. Without LB, a single server failure or traffic spike takes down the service.

Layer 4 (transport) balancers route by IP and TCP/UDP port — fast, no protocol awareness, can't route by URL path or HTTP headers. Examples: AWS NLB, HAProxy in TCP mode. Good for non-HTTP protocols and extreme throughput.

Layer 7 (application) balancers understand HTTP — route by hostname, path, headers, cookies. Can terminate TLS, inject headers, rewrite URLs. Examples: AWS ALB, NGINX, Envoy. Higher latency than L4 but far more routing flexibility.

Round-robin: equal distribution in rotation — simple but ignores server load. Least-connections: sends to backend with fewest active connections — better for long-lived requests. Weighted round-robin: more traffic to powerful nodes. IP hash: same client IP → same backend (sticky without cookies).

Health checks: active (LB probes /health endpoint periodically, removes unhealthy nodes) vs passive (mark unhealthy on request failure). Configure interval, timeout, healthy/unhealthy thresholds. Aggressive checks cause flapping; lenient checks route to dead backends.

Sticky sessions (session affinity) pin a user to one backend via cookie or IP hash. Needed for in-memory session state but breaks even load distribution and complicates deploys. Prefer external session store (Redis) and stateless backends.

Anycast routes the same IP to geographically nearest edge via BGP — used by CDNs and global LBs (Cloudflare, Google). Reduces latency without DNS round-robin. BGP health withdrawal removes bad PoPs from routing.

DNS load balancing returns multiple A records — clients pick randomly. TTL affects failover speed (low TTL = faster failover, more DNS load). DNS is not a true LB — no health awareness unless using smart DNS (Route53 health checks).

Architecture Diagram

Users / clients
         |
  Load Balancing
         |
  Core services
         |
  Data + observability

Examples

upstream api_pool {
    least_conn;
    server 10.0.1.10:8080 weight=3 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080 backup;
    keepalive 64;
}

server {
    listen 443 ssl;
    location /api/ {
        proxy_pass http://api_pool;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header X-Request-Id $request_id;
    }
}

backend web_servers
    balance leastconn
    option httpchk GET /health HTTP/1.1\r\nHost:\ health.local
    http-check expect status 200
    server web1 10.0.1.10:80 check inter 3s fall 3 rise 2
    server web2 10.0.1.11:80 check inter 3s fall 3 rise 2
    server web3 10.0.1.12:80 check inter 3s fall 3 rise 2

aws elbv2 create-target-group \
  --name api-tg --protocol HTTP --port 8080 \
  --vpc-id vpc-abc --health-check-path /health \
  --healthy-threshold-count 2 --unhealthy-threshold-count 3

aws elbv2 register-targets \
  --target-group-arn arn:aws:... \
  --targets Id=10.0.1.10 Id=10.0.1.11

Interview Questions

Best Practices

• Use L7 load balancing when you need path-based routing, TLS termination, or header inspection — L4 only when raw TCP throughput matters.
• Configure active health checks with thresholds — mark unhealthy after N failures, require M successes to rejoin.
• Enable connection keepalive between LB and backends — avoids TCP handshake overhead at high RPS.
• Set reasonable timeouts (connect, read, idle) per upstream latency profile — defaults cause false unhealthy marks.
• Avoid sticky sessions unless required — they break even load distribution and complicate rolling deploys.
• Terminate TLS at the LB and use HTTP/2 to backends where supported — reduces connection count.

Common Mistakes

• Round-robin to backends with different capacity — weak nodes get equal traffic and become the bottleneck.
• Health checks that hit a lightweight /health but app DB is down — LB routes traffic to broken instances.
• No connection draining on deploy — killing backends mid-request causes 502s during rollouts.
• IP-hash stickiness behind NAT — entire office maps to one backend, defeating load distribution.
• Single load balancer with no redundancy — LB itself becomes a single point of failure.

Trade-off Analysis

Load Balancing improves l4/l7 balancers, algorithms, health checks, sticky sessions. but adds operational and cognitive complexity — justify with load and team size.

Favor simplicity until metrics (p99 latency, error rate, cost) prove the pattern necessary.

Every redundancy layer trades capital/operational cost for availability — align with explicit SLO targets.

Document accepted inconsistency windows and recovery behavior before production cutover.

Cheat Sheet

Practical Exercises