Quick answer
Troubleshoot overload as a causal investigation, not a search for the busiest graph. Start with the page, confirm telemetry freshness, define the affected user journey and SLO symptom, and establish the change and demand timeline. Use RED evidence for requests—rate, errors, and duration—and USE evidence for constrained resources—utilization, saturation, and errors. Then connect offered demand to admission, in-flight work, queue age, pool wait, retries, and probe-driven capacity changes.
Write a falsifiable hypothesis before changing several controls. Prefer reversible mitigation that reduces new work or protects the bottleneck: freeze a rollout, pause optional producers, remove redundant retries, shed low-priority requests, reduce unsafe concurrency, or disable an explicitly safe optional feature. Do not blindly add workers, consumers, queues, or database connections; each can transfer collapse downstream.
Recovery is a measured phase. Drain only useful work, restore traffic and consumers gradually, and check user outcomes, resource pressure, backlog age, retry volume, and capacity membership after every step. A quiet alert, empty queue, or handful of successful traces is not enough. Close only after fresh evidence remains inside agreed thresholds for the observation window and ambiguous commerce outcomes are reconciled.
Shared flash-sale incident stage
A promotion creates checkout overload. Offered request rate exceeds tested sustainable capacity. The edge continues accepting work; in-flight order requests rise and order API latency breaches its objective. Inventory database pool wait grows, so requests spend their useful deadline before query execution. The queue backlog appears first as depth and then as oldest age, while oversized executor and broker buffers hide rejection.
Clients, the edge, and the order service retry the same failures, amplifying demand. Interactive checkout shares workers and connections with reconciliation. Readiness uses the constrained path and removes Pods; liveness restarts them. Cold replacements add connection churn and reduce warm capacity, sending more work to fewer endpoints. A pool-size increase would move pressure into the inventory database rather than create query capacity.
The incident commander applies an evidence-based recovery sequence. Confirm the page maps to a checkout SLO breach and that dashboards are current. Freeze the rollout, pause optional producers, remove duplicated retries, shed low-priority arrivals, preserve an interactive bulkhead, and expire work that cannot meet its deadline. Disable only fallbacks known to preserve payment, inventory, order, and authorization invariants.
Then ramp consumers and traffic in small steps. At each step compare good checkout events, tail latency, offered and admitted rate, rejection, in-flight work, oldest queue age, pool wait, retry volume, and readiness/liveness transitions. Keep the incident open until backlog age and ambiguous orders are resolved and capacity remains stable through a representative demand window.
Core mechanism and evidence boundary
Use one evidence sheet with six rows: user impact, demand, service work, waiting, dependency pressure, and effective capacity. Every row records metric definition, unit, scope, timestamp, freshness, baseline, current value, and owner. Align deploys, configuration changes, feature flags, caller releases, scheduled jobs, and traffic shifts on the same timeline. Correlation is a lead; a mechanism plus a falsifiable prediction is a hypothesis.
RED describes request behavior: offered and completed rate in requests per second, outcome counts, and latency in seconds. USE describes each resource: utilization ratio, saturation such as queued waiters, and error rate. SLO evidence states the valid event population and good-event rule. A checkout SLO should not silently count deliberate capacity rejection as success unless the product contract truly promises that result.
The shared dimensions are bounded. route is a low-cardinality dimension containing a bounded route template, never a raw path. operation is a low-cardinality dimension for stable named work such as order.create. priority is a low-cardinality dimension for a bounded class or tier. admission is a low-cardinality dimension for the accepted or denied capacity decision. rejection is a low-cardinality dimension for the denied-work decision with reviewed values such as queue_full.
deadline.remaining_ms is a teaching field in milliseconds and not a stable OpenTelemetry semantic convention. It remains only in protected traces and events. At the metric boundary, convert the value to seconds and observe a separately defined Prometheus histogram named backend_deadline_remaining_seconds; never expose a millisecond Prometheus variant for the same measurement. trace_id, request_id, and message_id are correlation fields in access-controlled traces and logs. Never use metric labels or metric dimensions for trace_id, request_id, message_id, order_id, user_id, a raw URL, exception text, a credential, a token, or PII.
The PromQL queries below use compatible route, operation, priority, admission, and rejection labels. Every selector also carries bounded dashboard placeholders for cluster, environment, service, and workload; replace them with exact reviewed values outside a dashboard. Every aggregation retains those four scope labels so unrelated targets cannot merge. Map the names explicitly if the platform uses equivalents such as deployment instead of workload. Counters end in _total, gauges represent current state, and durations use seconds. Recording rules may be preferable at high volume.
Page on sustained user impact, fast SLO burn, or an imminent actionable breach. A pressure-only signal with no user impact and no immediate exhaustion normally opens a ticket or warning for capacity investigation; it becomes a page when a documented threshold predicts near-term impact and an on-call action exists. Record which rule fired before querying.
# Offered, admitted, and rejected decisions per second.
sum by (cluster, environment, service, workload, route, operation, priority, admission, rejection) (
rate(backend_admission_decisions_total{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload", route="/checkout"
}[5m])
)
# Completed request rate per second, split by bounded outcome.
sum by (cluster, environment, service, workload, route, operation, outcome) (
rate(backend_requests_completed_total{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload", route="/checkout"
}[5m])
)
# Request error rate per second.
sum by (cluster, environment, service, workload, route, operation) (
rate(backend_requests_completed_total{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload",
route="/checkout", outcome="error"
}[5m])
)
# Request error ratio: error completions divided by all completions; unitless.
sum by (cluster, environment, service, workload, route, operation) (
rate(backend_requests_completed_total{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload",
route="/checkout", outcome="error"
}[5m])
)
/
sum by (cluster, environment, service, workload, route, operation) (
rate(backend_requests_completed_total{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload", route="/checkout"
}[5m])
)
# Checkout SLO good/valid ratio; both sides are events per second.
sum by (cluster, environment, service, workload, journey) (
rate(backend_slo_events_total{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload",
journey="checkout", valid="true", good="true"
}[5m])
)
/
sum by (cluster, environment, service, workload, journey) (
rate(backend_slo_events_total{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload",
journey="checkout", valid="true"
}[5m])
)
# Burn rate: observed bad-event ratio divided by allowed bad-event ratio.
# $slo_objective is the configured scalar objective, for example from the SLO rule.
(
1 -
(
sum by (cluster, environment, service, workload, journey) (
rate(backend_slo_events_total{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload",
journey="checkout", valid="true", good="true"
}[5m])
)
/
sum by (cluster, environment, service, workload, journey) (
rate(backend_slo_events_total{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload",
journey="checkout", valid="true"
}[5m])
)
)
)
/
(1 - $slo_objective)
# p95 request duration in seconds from a classic histogram.
histogram_quantile(
0.95,
sum by (cluster, environment, service, workload, le, route, operation) (
rate(backend_request_duration_seconds_bucket{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload", route="/checkout"
}[5m])
)
)
# Current active work, a request-count gauge.
sum by (cluster, environment, service, workload, operation, priority) (
backend_in_flight_requests{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload"
}
)
# Queue depth is a work-item gauge; oldest age is seconds.
sum by (cluster, environment, service, workload, operation, priority) (
backend_queue_depth{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload"
}
)
max by (cluster, environment, service, workload, operation, priority) (
backend_queue_oldest_age_seconds{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload"
}
)
# Inventory pool USE utilization: in-use connections / capacity; unitless.
sum by (cluster, environment, service, workload, operation) (
backend_pool_connections{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload",
operation="inventory.reserve", state="in_use"
}
)
/
sum by (cluster, environment, service, workload, operation) (
backend_pool_capacity_connections{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload",
operation="inventory.reserve"
}
)
# Inventory pool USE saturation: current waiter-count gauge.
sum by (cluster, environment, service, workload, operation) (
backend_pool_waiters{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload",
operation="inventory.reserve"
}
)
# Inventory pool USE errors: failed acquisitions per second.
sum by (cluster, environment, service, workload, operation) (
rate(backend_pool_acquire_errors_total{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload",
operation="inventory.reserve"
}[5m])
)
# p95 inventory pool wait in seconds.
histogram_quantile(
0.95,
sum by (cluster, environment, service, workload, le, operation) (
rate(backend_pool_wait_seconds_bucket{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload",
operation="inventory.reserve"
}[5m])
)
)
# p10 remaining deadline in seconds at the metric boundary.
histogram_quantile(
0.10,
sum by (cluster, environment, service, workload, le, route, operation) (
rate(backend_deadline_remaining_seconds_bucket{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload", route="/checkout"
}[5m])
)
)
# Additional retry attempts per second.
sum by (cluster, environment, service, workload, operation) (
rate(backend_retry_attempts_total{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload"
}[5m])
)
# Probe state transitions per second; inspect restarts separately.
sum by (cluster, environment, service, workload, probe, result) (
rate(backend_probe_transitions_total{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload"
}[5m])
)
sum by (cluster, environment, service, workload, namespace) (
increase(backend_container_restarts_total{
cluster="$cluster", environment="$environment",
service="$service", workload="$workload"
}[15m])
)Run each commented expression separately after substituting bounded dashboard values. The completed and error queries have requests-per-second units; their ratio is unitless. The SLO numerator counts events that are both valid and good, while its denominator counts every valid event. Burn rate divides the observed bad-event fraction by the allowed fraction from the configured objective. A zero or absent denominator yields no finite ratio; treat that as no valid population or missing telemetry and alert on telemetry health separately, never as user failure. Pool utilization is a ratio, saturation is current waiters, pool errors and retries are events per second, queue depth is work items, and wait/age/deadline histograms are seconds. Use complete counters and gauges for denominators. Sampled traces explain selected queue waits or exhausted deadlines; they cannot establish concurrency, rejection rate, or population latency. Validate scrape and export health before trusting a zero. Compare queue depth with oldest age and throughput; consumer prefetch and active work can make a broker queue look deceptively small.
Minimal reproducible implementation
Follow this incident procedure and write each result into the evidence sheet:
1. PAGE
acknowledge; assign incident commander, operations lead, and communications owner
record page time, alert expression, evaluation window, and telemetry freshness
2. IMPACT
name user journey, regions, tenants or priority classes, first bad time
record SLO good/valid events, error-budget burn, rate, errors, and duration
3. TIMELINE
place deploys, config, flags, producer jobs, traffic, dependency events,
readiness changes, and restarts on one clock
4. PRESSURE
compare offered/admitted/rejected/completed rates
inspect in-flight, queue depth plus oldest age, pool wait, retries,
CPU/memory/threads/connections, and effective ready capacity
5. HYPOTHESIS
state mechanism, supporting and contradicting evidence, and prediction
choose one reversible mitigation that tests the prediction safely
6. MITIGATION
freeze risky change; pause optional load; remove retry amplification;
shed low priority; lower unsafe concurrency; preserve invariants
timestamp every action and watch the predicted leading and user signals
7. RECOVERY
expire obsolete work; reconcile ambiguous writes; ramp load in stages
hold each stage for the agreed window; reverse if pressure or impact returns
8. CLOSE
verify SLO, latency, in-flight, rejection, queue age, pool wait, retries,
probes, restarts, backlog disposition, and telemetry freshnessFor example, hypothesize: “A new order concurrency target raised active inventory calls above the pool’s safe envelope; lowering that target will reduce pool wait and request duration without lowering completed checkout throughput.” Supporting evidence is a target change followed by in-flight and pool-wait growth. Contradicting evidence would be flat inventory concurrency or an earlier database latency shift. The prediction names the signals and direction before the change.
If the safe action is to reduce the limit, do so in one reviewed step and observe. If pool wait falls while completions remain steady and the SLO improves, the evidence supports the mechanism. If completions collapse or wait is unchanged, reverse or revise. Avoid changing pool size, replicas, retry policy, and concurrency simultaneously unless immediate safety requires a broad emergency action; document every emergency change for later controlled separation.
Failure modes and dangerous misconceptions
“CPU is high, so add replicas.” CPU can be productive work or retry overhead, and replicas may overload a shared database. Establish the constrained resource and scaling dependency before acting.
“Queue depth is falling, so recovery is done.” Work may have expired, moved into consumer prefetch, or failed. Check oldest age, admitted and completed counts, disposition, user outcomes, and downstream pressure.
“More database connections reduce pool wait.” They can increase database concurrency, locks, memory, and tail latency. Raise a pool only after proving downstream headroom under a tested workload.
“Every error needs a retry.” Retrying overload at multiple layers multiplies work. Preserve one bounded attempt budget, idempotency, jitter, and remaining deadline, and stop retrying permanent or unsafe outcomes.
“Readiness will drain overload safely.” Removing endpoints concentrates traffic and has propagation delay. Readiness is an instance-routing signal, not per-request admission or a substitute for load shedding.
“A trace proves the cause.” A trace supports an execution story, but sampling and selection prevent it from defining population rates. Validate the hypothesis with complete metrics and controlled action.
“The alert cleared.” An alert can resolve while backlog, stale fallback, ambiguous orders, retry waves, or probe churn remain. Use explicit recovery gates and an observation window.
“Restore everything at once.” Synchronized producers, consumers, clients, and cold caches can recreate the incident. Ramp one pressure source at a time and keep a reversal path.
Security/privacy/capacity/cost implications
Incidents invite broad access and ad hoc logging. Preserve least privilege, audit emergency actions, redact payloads, and keep customer identifiers and secrets out of dashboards and chat. Use protected correlation records for individual order reconciliation. Do not disable authentication or authorization to reduce latency; use a tested safe fallback or fail closed.
Attack traffic, oversized bodies, expensive parsing, and forged priority can resemble organic overload. Check authentication outcomes, tenant distribution, request size, cache behavior, and edge controls using bounded dimensions. Keep priority classification server-owned and preserve reserved operational access without exposing an unaudited bypass.
Mitigation moves cost. Extra replicas add compute but may multiply database pressure; larger queues add memory or broker storage and prolong recovery; verbose telemetry adds ingestion and query load. Record temporary capacity and diagnostics with owners and expiry. After recovery, convert the measured bottleneck, failover envelope, retry demand, and warm-up rate into a capacity plan rather than leaving emergency overprovisioning unexplained.
Testing and production validation
Rehearse the runbook with a flash-sale load profile, one slow inventory dependency, one lost replica, retrying callers, background consumers, and probe contention. Start from a healthy baseline and verify every dashboard’s units, labels, freshness, and link to raw evidence. Inject failures one at a time and then in the shared causal chain.
Rehearse partial observability failure too. Delay one metrics pipeline, drop a dashboard query, reset a counter through restart, and make a trace sample unrepresentative. Responders should mark uncertainty, inspect scrape and export health, retain the exact query and evaluation window, and use an independent user or control-plane signal before interpreting a zero as recovery. Align clocks and event timestamps before ordering deploys, probe transitions, and demand changes. If evidence is incomplete during active impact, choose the mitigation with the safest bounded downside and state which observation would trigger reversal. After telemetry returns, reconstruct the interval and verify that the action changed the predicted mechanism rather than merely changing what was visible.
Assert that responders can distinguish offered from admitted work, depth from age, pool utilization from pool wait, original attempts from retries, readiness transition from restart, and degraded success from failed invariants. Test ambiguous payment and order outcomes with idempotency and reconciliation. Confirm logging remains redacted during debug escalation.
Run recovery drills, not only failure drills. Expire obsolete work, ramp consumers, restore retries and optional features, and return concurrency to its reviewed target in stages. At each hold point verify SLO, RED, USE, backlog, dependency, and capacity membership. Feed observed safe rates and detection gaps into dashboards, alerts, limits, and the post-incident action list.
Operations checklist
- Confirm the page, telemetry freshness, affected journey, SLO, and error-budget burn.
- Build one timeline for demand, deploys, configuration, dependencies, and probe transitions.
- Compare offered, admission, rejection, completed, expired, and retry volume.
- Inspect in-flight work, bulkhead occupancy, queue depth and oldest age, and pool wait.
- State a falsifiable overload hypothesis and predicted signal movement.
- Apply reversible load shedding or producer pauses before adding downstream pressure.
- Verify the end-to-end deadline, cancellation handling, and one bounded retry budget.
- Preserve each bulkhead, bounded queue, and fail-closed fallback invariant.
- Treat readiness as a routing signal; inspect liveness restarts and effective capacity.
- Ramp recovery, reconcile ambiguous writes, hold the observation window, and verify the user SLO.
Official sources
- Google SRE: Handling Overload
- Google SRE: Addressing Cascading Failures
- Prometheus instrumentation practices
- Prometheus querying basics
- OpenTelemetry metrics semantic conventions
- Kubernetes probes documentation
Accessed 2026-08-02. Query names are a coherent teaching contract for this cluster; map them explicitly to the versioned instrumentation and Kubernetes environment actually deployed.
Related reading
Use Production Resilience for Backend Systems as the architecture frame, Backpressure, Bounded Queues, and Flow Control for backlog evidence, Load Shedding and Adaptive Concurrency Limits for admission, and Liveness, Readiness, and Startup Health Checks for capacity transitions. Continue through System Design or browse Topics.
If overload evidence points toward database work, Database Query Performance for Backend Systems separates admission and pool pressure from execution and waits; Production Slow Query Troubleshooting provides the focused plan, statistics, lock, maintenance, mitigation, and recovery checks needed before restoring full traffic.