Production Incident Troubleshooting with Logs, Metrics, and Traces

Quick answer

Production troubleshooting is a sequence of evidence-based decisions. Start with user impact and incident scope, not a random log search. Use the SLO alert to identify the affected journey and urgency. Use metrics to compare services, routes, outcomes, resources, and time. Use a representative trace to identify the dependency or queue boundary where duration accumulates. Use structured logs to recover detailed facts. Form a falsifiable hypothesis, choose the safest mitigation, and verify recovery with fresh user and component evidence.

Logs, metrics, and traces are complementary. Metrics are efficient for scope and trends. Traces connect one execution across services. Logs preserve detailed events. Deployment and configuration events explain what changed. Every conclusion should name the signal, time range, population, and uncertainty. Sampling, delayed export, missing telemetry, and high-cardinality filtering can all distort evidence.

The incident is not complete when a graph turns green once. Recovery verification confirms SLO burn, latency, retry volume, connection pool wait, queue backlog, and telemetry freshness over an appropriate window. A postmortem then records impact, timeline, contributing conditions, detection gaps, response decisions, and durable actions without blaming individuals.

Learning objectives

Run an evidence-driven Spring order service incident from user-impact scope through reversible mitigation and verified recovery.
Correlate telemetry quality, logs, metrics, traces, JFR, changes, dependencies, and durable outcomes without treating one signal as root-cause proof.

Prerequisites

Complete Incident Command, Communication, and Postmortems and understand the preceding signal and SLO lessons.

Production failure scenario

The shared incident begins when a slow query in the inventory service exhausts its database connection pool.

At 14:02 UTC, a multi-window burn-rate alert pages for checkout latency. Traffic rate is normal, overall errors are only slightly elevated, but good events within 750 milliseconds have fallen. At 13:54 a deployment changed an inventory reservation query. The query now scans more rows for one product class and holds database connections longer.

Inventory pool utilization approaches its limit. Pending acquisition and pool wait rise. The order API times out and retries, amplifying inventory traffic. Reservation consumers use the same pool, so processing slows and queue backlog grows. Notifications are delayed. The user-visible symptom is checkout latency; the primary mechanism is pool saturation caused by slow database work; retries and shared resource contention amplify the incident.

Resource identity uses service.name and deployment.environment.name. Aggregate metrics use bounded route, status, and operation. A representative trace supplies trace_id and span_id; support and application logs carry request_id; asynchronous reservation evidence carries stable message_id. These fields support pivots but never become metric labels or authorization claims.

Evidence and system boundary

An incident timeline mixes facts and interpretations. A deployment event at 13:54 and burn alert at 14:02 are facts. “The deployment caused the incident” is a hypothesis until rollback or comparison supports it. Record confidence and disconfirming evidence.

OpenTelemetry semantic conventions help compare services, but an attribute’s existence does not prove instrumentation completeness. W3C trace context supports propagation, but sampled or dropped spans can make a trace incomplete. Prometheus rates and histograms depend on query windows, labels, scrape health, and bucket design. Logs can be delayed or dropped by the pipeline.

Use independent signals where possible. A user SLI and synthetic check can establish impact. RED and USE metrics locate service and resource pressure. Traces explain causal shape. Logs provide query name, pool wait, attempt, and error classification. Collector internal telemetry verifies that missing data is not merely export failure.

Minimal implementation

Use a runbook with explicit decision gates.

1. Acknowledge and establish impact

Record incident owner, communications lead, start time, alert, affected journey, environment, and current SLO burn. Confirm the page is based on fresh data. Check synthetic and support signals. If there is no verified impact, continue investigation without declaring recovery or broadening access unnecessarily.

2. Bound the time and population

Compare before, during, and current windows. Slice by service, route template, region, version, and bounded outcome. Do not group by raw URL, user ID, order ID, exception message, or trace_id. Determine whether all checkout requests, one product class, or asynchronous reservations are affected.

3. Follow symptom to resource

Use RED metrics:

histogram_quantile(
  0.95,
  sum by (service, route, le) (
    rate(http_server_request_duration_seconds_bucket{environment="production"}[5m])
  )
)

Then apply USE to inventory connections:

sum(db_client_connection_pool_usage{service="inventory-service",state="used"})
/
sum(db_client_connection_pool_limit{service="inventory-service"})

Check pending requests, acquisition timeout rate, retry attempts, queue depth, and oldest-message age. Align them with deployment events.

4. Inspect a representative trace and logs

Choose an exemplar or trace from the affected latency bucket. Confirm that time accumulates in pool wait or database operation rather than guessing from the top-level span. Pivot to logs by trace_id, verify service.name, deployment version, normalized operation, query name, pool wait, and attempt. For delayed messages, use message_id to examine redelivery and processing attempts.

5. State and test a hypothesis

Write: “The new inventory query increases connection hold time for product class X; this saturates the shared pool, causes order retries, and delays reservation consumers.” Predict what would differ in an unaffected class, previous version, or replica. Run a safe read-only comparison. A hypothesis that cannot be disproved is not operationally useful.

6. Mitigate safely

Prefer reversible action: roll back the query change, disable the affected path, reduce retry amplification, or shift traffic according to established controls. Do not increase pool size blindly; the database may receive more concurrent expensive queries. Record the change, owner, expected signal movement, and rollback condition.

7. Verify recovery

Require fresh evidence: checkout good-event ratio, both burn-rate windows, inventory duration, pool wait, retries, queue age, and exporter freshness. Confirm no new region or route regresses. Continue watching until backlog drains and delayed work finishes within its freshness objective.

8. Communicate with evidence

Publish concise updates on a fixed cadence: user impact, affected scope, current hypothesis, mitigation in progress, measurable response, and next checkpoint. Separate confirmed facts from investigation. Avoid pasting sensitive logs or customer identifiers. If the estimated recovery time is unknown, say what evidence will make it knowable. Clear updates reduce duplicate work and prevent stakeholders from treating a speculative cause as settled.

9. Preserve a decision timeline

Record alerts, ownership changes, deploys, queries, traces, mitigation commands, approvals, and observed results with timestamps. Link durable dashboards or saved queries where retention permits. The timeline should let a reviewer reconstruct why an action was reasonable with the information available then. It is not a transcript of every chat message and should not contain secrets.

10. Transition from response to follow-up

When recovery criteria hold, declare the response phase complete, but keep backlog and delayed effects under observation. Schedule the postmortem, identify data that will expire, and create immediate safety tasks separately from long-term redesign. Restore temporary debug settings and break-glass access. A mitigation such as rollback is not automatically the permanent fix.

The postmortem should distinguish trigger, contributing conditions, and systemic weaknesses. The query change may trigger the incident; shared connection pools, aggressive retries, missing query-plan tests, slow burn-rate detection, or incomplete queue freshness objectives may enlarge it. Actions should have owners, deadlines, and verification. “Be more careful” is not an actionable control.

Failure modes and trade-offs

Starting with logs encourages confirmation bias. Searching for “timeout” returns familiar errors even when the dominant issue is saturation. Start with scope and comparison, then use logs for detail.

Changing several variables destroys causal evidence. Rolling back code, increasing pool size, and scaling consumers simultaneously may recover the service but conceal which action mattered and create new database load. In severe impact, safety outranks perfect experimentation, but record each action and timestamp.

Retries are often treated as mitigation when they are amplification. If the dependency is saturated, more attempts increase contention. Coordinate retry reduction with idempotency and user behavior. Do not disable correctness safeguards merely to improve a graph.

Sampling can hide rare or normal traces. Tail policies may retain only slow and failed requests, which is useful for diagnosis but unsuitable for estimating rates. Use metrics for population claims and state the selection criteria for any trace example.

Telemetry delay can produce false recovery or false recurrence. Check Collector queues, exporter failures, backend ingestion lag, and source timestamps. Compare custom dashboards with direct queries when caching is suspected.

Common misconceptions

Restarting, scaling, or rolling back successfully is not recovery proof.
The loudest log message is not automatically the cause.
One retained trace or JFR recording is not population evidence.
A cleared alert can coexist with delayed telemetry or ambiguous durable orders.
Incident speed does not justify exposing secrets, customer data, or unrestricted queries.

Security, privacy, and cost

Incidents create pressure to broaden access and log more data. Keep least privilege. Use approved break-glass procedures, time-bounded access, and auditing. Do not paste tokens, customer records, raw payloads, or sensitive traces into chat channels or tickets. Redact screenshots and exported queries.

Correlation IDs are lookup tools, not customer identifiers or credentials. Validate request_id, trace_id, span_id, and message_id before using them in queries. Do not convert them to metric labels. External baggage remains untrusted.

Emergency debug logging can multiply cost and expose secrets. Scope it to service, duration, and safe fields; set an automatic expiry; monitor volume; and verify removal. Preserve relevant evidence under the correct retention policy without copying entire datasets indefinitely.

Testing and validation

Run game days against a non-production environment with the same instrumentation contract. Inject query latency, constrain a pool, trigger retries, and slow a consumer. Verify the SLO detects impact, RED and USE locate the mechanism, traces show pool wait, logs preserve normalized details, and queue age captures delayed work.

Test missing evidence. Drop trace context, disable one exporter, delay logs, and remove one metric series. The runbook should detect telemetry failure rather than interpret absence as health. Test sampled traces and prove population claims still come from complete metrics.

Validate alert routing, ownership, links, and permissions. Ensure the on-call can query the necessary systems without permanent administrator access. Measure time to impact confirmation, first useful hypothesis, mitigation, and recovery verification.

After a real incident, replay the timeline with stored queries. Confirm the postmortem actions address causes or detection gaps: query-plan regression tests, pool isolation, retry budgets, backlog objectives, deployment annotations, or pipeline freshness alerts. Track actions to completion with owners and dates.

Evaluate the runbook against two competing hypotheses. For example, database pool saturation and an unavailable trace backend can both produce missing spans, but only the first should raise pool wait and user latency. The responder must compare independent signals before changing the application. This practice reduces fixation on the first plausible graph.

Test handoff between responders. The incoming owner should reconstruct current impact, hypothesis, actions, and next decision from the timeline without a private verbal briefing. If critical context exists only in one person’s terminal history, incident state is not durable.

Measure recovery lag for asynchronous work. HTTP latency may normalize immediately after rollback while old reservation messages remain queued. Keep queue freshness, dead-letter volume, and consumer error disposition visible until the backlog reaches a known safe state. Avoid scaling consumers beyond database capacity and recreating saturation.

Finally, test communications failure. If the normal chat, dashboard, or identity provider is degraded, responders need documented alternatives and read-only evidence access. This does not justify permanent broad credentials; it requires reviewed break-glass paths, expiration, and audit.

Define stop conditions for investigation branches. If a query, trace, or log search cannot distinguish hypotheses within a bounded time, return to the evidence map rather than deepening an unproductive search. Escalate to the component owner with the facts already established. This preserves responder attention and makes parallel work explicit. The incident lead should close duplicate investigations and keep one authoritative hypothesis list with evidence for and against each item.

After closure, update the runbook with validated queries and remove temporary shortcuts. A runbook that accumulates obsolete commands becomes dangerous. Assign an owner and review it after telemetry, service, or deployment changes.

Practice the sequence often enough that responders know where evidence lives but still verify current links and freshness. Familiarity should shorten navigation, not turn an old dashboard interpretation into an unquestioned assumption.

Place those investigation steps inside the prevention, containment, degradation, and recovery model from Production Resilience for Backend Systems. When saturation is the leading hypothesis, use Production Overload Troubleshooting to coordinate admission, deadlines, bulkheads, queues, fallbacks, readiness, and verified exit criteria.

Decision checklist

Acknowledge the alert and assign clear incident roles.
Verify user impact and telemetry freshness.
Bound time, environment, route, region, version, and operation.
Use metrics for scope, traces for causality, and logs for detail.
Record facts separately from hypotheses.
Prefer reversible, low-blast-radius mitigation.
Timestamp every action and expected outcome.
Watch retry amplification and shared-resource pressure.
Verify recovery across SLO, component, backlog, and pipeline signals.
Publish a blameless postmortem with owned durable actions.

Official sources

Official sources accessed August 19, 2026. Adapt incident roles and escalation thresholds to the organization’s documented response policy.

For framework-specific order-service failures, Production Spring Boot Incident Troubleshooting applies this loop to version and configuration drift, MVC and security boundaries, transaction outcomes, JPA query evidence, pool and thread pressure, Actuator signals, and idempotent recovery.

Previous: Incident Command, Communication, and Postmortems. Next: SLO Incident Triage Lab Examples. Follow Production Observability & SRE and its topic.

Use SLI, SLO, Error Budgets, and Burn-Rate Alerts for detection and Structured Logging and Correlation IDs for detailed pivots. Review the full sequence in System Design and Topics.

For database incidents, Database Query Performance for Backend Systems defines the application-to-engine evidence boundary, and Production Slow Query Troubleshooting supplies the specialized sequence for normalized query families, plans, waits, maintenance, mitigations, and workload-wide recovery.