Microservices
Service boundaries, API contracts, operational complexity.
Theory
Microservices decompose an application into independently deployable services bounded by business domains (DDD). Each service owns its data, codebase, and release cycle — enabling team autonomy and technology heterogeneity at the cost of distributed system complexity.
Decompose by domain capability (orders, payments, inventory) not by technical layer (all DBs together). Services communicate via well-defined APIs. Avoid distributed monolith — services that must deploy together defeat the purpose.
Synchronous communication (REST, gRPC) is simple and request-response oriented — caller blocks until response. Good for queries needing immediate answers. Creates runtime coupling: callee failure propagates to caller without circuit breakers.
Asynchronous communication (message queues, event buses) decouples producers and consumers in time. Events announce facts ("OrderPlaced") — multiple subscribers react independently. Enables eventual consistency and load leveling but harder to debug and trace.
Distributed transactions across services are avoided. Saga pattern coordinates local transactions with compensating actions on failure (e.g. cancel payment if shipment fails). Choreography (events) vs orchestration (central coordinator) — choreography scales better; orchestration is easier to reason about.
Two-phase commit (2PC) provides atomicity across databases but blocks on coordinator failure and doesn't scale — rarely used in microservices. Prefer sagas with idempotent consumers and outbox pattern for reliable event publishing.
Service mesh (Istio, Linkerd) injects sidecar proxies handling mTLS, retries, timeouts, traffic splitting, and observability without app code changes. Circuit breaker pattern stops calling failing services after threshold — fail fast instead of cascading timeouts.
Distributed tracing (OpenTelemetry, Jaeger, Zipkin) correlates requests across services via trace IDs propagated in headers. Essential for debugging latency in microservice call chains. Without tracing, a 2s API response hiding five 400ms serial calls is invisible.
Architecture Diagram
Users / clients
|
Microservices
|
Core services
|
Data + observabilityExamples
// order.proto
service OrderService {
rpc CreateOrder(CreateOrderRequest) returns (Order);
rpc GetOrder(GetOrderRequest) returns (Order);
}
// Client calls with deadline
ctx, cancel := context.WithTimeout(ctx, 500*time.Millisecond)
defer cancel()
order, err := client.CreateOrder(ctx, &pb.CreateOrderRequest{...})
// Order service publishes
publish("OrderPlaced", { orderId, amount })
// Payment service consumes, publishes
on("OrderPlaced") -> charge() -> publish("PaymentCompleted")
on("PaymentFailed") -> publish("PaymentFailed")
// Order service compensates
on("PaymentFailed") -> cancelOrder(orderId)
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
name: payment-cb
spec:
host: payment-service
trafficPolicy:
connectionPool:
tcp: { maxConnections: 100 }
http: { http1MaxPendingRequests: 50 }
outlierDetection:
consecutiveErrors: 5
interval: 30s
baseEjectionTime: 60s
Interview Questions
How do you decide service boundaries in microservices?
Align with business domains (bounded contexts in DDD). Each service owns one capability and its data. Services should change together rarely. If two services always deploy together, merge them. Start with a modular monolith and split when scaling or team boundaries demand it.
REST vs gRPC for inter-service communication?
REST: HTTP/JSON, human-readable, browser-friendly, larger payloads. gRPC: HTTP/2, Protobuf binary, strongly typed, streaming, lower latency — ideal for internal service-to-service. Use gRPC internally, REST/GraphQL at the edge for clients.
What is the saga pattern and when do you use it?
Sequence of local transactions with compensating actions on failure — no distributed lock. Choreography: services react to events. Orchestration: central saga manager coordinates steps. Use when business transactions span services and 2PC is unacceptable.
What does a circuit breaker do?
After N consecutive failures to a downstream service, the breaker opens — fail fast without waiting for timeout. After cooldown, half-open state tests one request. Prevents thread exhaustion from cascading timeouts. Implement in app code (resilience4j) or service mesh (Istio outlier detection).
What is the outbox pattern?
Write business data and outbound event to the same local DB transaction (outbox table). Separate relay process publishes events to message broker and marks outbox rows sent. Guarantees at-least-once event delivery without dual-write inconsistency between DB and queue.
What problems does a service mesh solve?
mTLS between all services, retries/timeouts, traffic splitting (canary), observability (metrics/traces per call), and policy enforcement — without modifying application code. Cost: sidecar resource overhead and operational complexity. Worth it at 20+ services with strict security requirements.
FAANG: When should you NOT use microservices?
Small team (<10 engineers), unproven product, no DevOps maturity, or when domain boundaries are unclear. Premature decomposition creates distributed monolith — all the pain of microservices without autonomy benefits. Start monolith/modular monolith; split on proven scaling or organizational drivers.
How do you debug latency across 10 microservices?
Distributed tracing with propagated trace context (W3C traceparent header). Each span records service name, operation, duration. Jaeger/Tempo UI shows critical path. Combine with structured logs including trace_id. Without this, serial 200ms calls across 8 services appear as one slow API.
Best Practices
- Draw service boundaries around team ownership and data — not around technical layers (never a "database service" for everything).
- Prefer async events for cross-domain workflows — synchronous chains of 5+ services create fragile latency multiplication.
- Implement circuit breakers and timeouts on every outbound call — default infinite wait hangs thread pools cluster-wide.
- Use idempotency keys on write APIs — retries are inevitable in distributed systems.
- Adopt distributed tracing (OpenTelemetry) from day one — debugging cross-service failures without trace IDs is guesswork.
- Design for independent deployability — if two services must ship together, they are not separate services yet.
Common Mistakes
- Splitting monolith by technical layer (API service, DB service) — creates chatty network calls without domain isolation.
- Synchronous chains for checkout/payment — one slow service blocks the entire user request path.
- No schema versioning on events — producers upgrade and break all consumers simultaneously.
- Shared database between services — couples schemas and prevents independent deployment.
- Distributed monolith — microservices repo structure but everything deployed as one unit.
Trade-off Analysis
Microservices improves service boundaries, api contracts, operational complexity. but adds operational and cognitive complexity — justify with load and team size.
Favor simplicity until metrics (p99 latency, error rate, cost) prove the pattern necessary.
Every redundancy layer trades capital/operational cost for availability — align with explicit SLO targets.
Document accepted inconsistency windows and recovery behavior before production cutover.
Cheat Sheet
Practical Exercises
Take a monolith module (e.g. notifications). Move to a separate service with its own DB. Communicate via HTTP or events. Document the API contract.
Model order → payment → inventory as three services. Implement cancel/refund compensating actions when inventory fails after payment.
Add OpenTelemetry to two services. Make a call chain A → B. View the trace in Jaeger and identify span latency breakdown.