Quick answer
Production Spring Boot troubleshooting is an evidence-guided recovery loop: stabilize users and durable correctness, identify which version and configuration are running, classify the failed boundary, apply the smallest reversible mitigation, and verify recovery in user, correctness, and resource signals. Do not begin with a restart, a pool increase, a database change, or a rollback because a dashboard looks alarming. Each action changes evidence and can widen the incident.
For the order service, collect a bounded timeline: deployment version, configuration fingerprint, traffic and error change, request validation and authorization outcomes, HTTP timeout location, active and waiting database connections, query/lock evidence, transaction outcomes, JPA query count or latency, security denials, Actuator readiness/liveness, and thread/pool state. Then ask what is true at the durable boundary. A client timeout does not tell us whether an order committed; a green JVM does not tell us whether it can reserve inventory; an exception in logs does not tell us whether the transaction rolled back.
This runbook uses Java 17 and Spring Boot 4.1 as its documented mainline. A Spring Boot 3.5 service should preserve the same evidence model while checking its exact Actuator properties, management endpoint configuration, pool implementation, JPA provider, security chain, and deployment platform. An incident is the worst time to guess an upgrade default.
Shared order-service incident stage
The shared flash-sale incident begins after a normal deployment when checkout p95 latency rises, some customers receive a gateway timeout, database pool wait grows, and inventory conflicts increase. Validation rejection and authorization denial rates are stable. Liveness is green; readiness begins to fail on several replicas. This is a useful starting observation, not a diagnosis. The order service could be slow because of an inefficient JPA query, lock contention on a hot SKU, a reduced connection limit, a remote call inside a transaction, a thread pool backlog, a configuration change, or a retry storm.
First protect users and correctness. Stop or reduce optional traffic, apply an admission limit to new checkout commands if the service cannot meet its deadline, preserve idempotency keys, and publish an honest retry message only where the command has not been accepted. Do not tell customers to submit a new order if the prior command may have committed. Maintain access to a protected status channel and a support-safe lookup that can report a recorded order outcome without exposing other customers.
Next build a timeline around the first symptom. Compare current version and configuration fingerprint with the last known healthy version. Split requests by safe route template and outcome family. Compare connection active/pending counts, pool wait, transaction duration, lock waits, slow-query evidence, CPU, GC, request executor queue, remote client latency, and outbox age. Correlate protected traces only by opaque request ID. The goal is to distinguish a saturated resource, a correctness conflict, and an application regression before modifying the system.
Core mechanism and evidence boundary
Start at the HTTP boundary. Confirm ingress and application request volume, status families, latency percentiles, request size, validation failures, and authentication/authorization denials. A jump in 400 responses may be a client contract deployment; a jump in 401/403 may be a credential, clock, issuer, or policy change; a jump in 5xx or gateway timeouts needs a boundary location. Ensure route labels are templates, not raw identifiers, and do not log full bodies or authorization headers.
Then verify version and configuration. Identify the immutable build or image digest, deployment time, replica set, Java version, active profile, feature flags, relevant timeout/pool settings, and configuration source revision. Use a sanitized configuration fingerprint rather than printing secrets. Configuration precedence matters: a command-line override, environment variable, mounted file, or platform injection can change a timeout or datasource setting even when the application artifact did not change. Compare effective values to a known-good record, not just source control.
At the application boundary, classify errors by where the response was committed. A validation exception before service execution differs from an exception after a transaction starts. A controller advice may format many application errors but cannot rewrite a response already committed by a filter, streaming body, reverse proxy, or client disconnect. Record whether an idempotency key was accepted, whether an order row exists, its state, and whether a reservation/outbox record committed before advising a customer to retry.
At the resource boundary, inspect pool wait before increasing pool size. High pending connections with long transaction duration often indicates slow queries, locks, remote calls held inside transactions, or leaked connections. Increasing the pool can increase active lock contenders and database load. Inspect a bounded sample of slow query fingerprints, query count, JPA fetch patterns, transaction duration, database locks, and connection acquisition time. N+1 behavior may turn a small query count increase into a pool-saturating fan-out; a single hot inventory row may create a lock queue even when average CPU is low.
At the execution boundary, distinguish request executor starvation from database saturation. Thread dumps are sensitive snapshots, not public diagnostics: collect them under access control, compare multiple samples, and look for threads waiting on pool acquisition, locks, remote I/O, or a bounded queue. A large executor queue can reflect offered load greater than capacity, a blocking downstream dependency, or too many retries. It is not a reason to create unlimited threads, which adds context switching, memory pressure, and even more database contention.
State an explicit decision threshold before acting. For example: if pool acquisition wait and lock wait rise together only for the hot SKU while query latency elsewhere remains normal, apply a narrowly scoped admission policy and investigate the reservation contention; if every route has pool wait after a configuration revision reduced the maximum size, restore the reviewed setting and watch connection/error recovery; if only one canary version has new 5xx evidence, stop that rollout before changing database capacity. These are hypotheses to test against data, not permanent rules.
Minimal reproducible Spring example
The following Java 17 sketch is not an automatic incident fixer. It shows an explicit, bounded diagnostic classification at the service boundary. Operators should expose only aggregated metrics publicly and protect logs or trace details. Spring Boot 4.1 and Spring Boot 3.5 deployments must confirm their own datasource, transaction manager, and Actuator configuration.
package com.example.orders;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
public class CheckoutService {
private final InventoryRepository inventory;
private final MeterRegistry metrics;
public CheckoutService(InventoryRepository inventory, MeterRegistry metrics) {
this.inventory = inventory;
this.metrics = metrics;
}
@Transactional(timeout = 3)
public CheckoutResult reserve(CheckoutCommand command) {
if (!inventory.reserveIfAvailable(command.sku(), command.quantity())) {
metrics.counter("orders.reservation.attempts", "result", "inventory_conflict").increment();
return CheckoutResult.conflict();
}
metrics.counter("orders.reservation.attempts", "result", "reservation_requested").increment();
return CheckoutResult.reserved();
}
}The class and transactional entry point are public and non-final so the ordinary class-based Spring proxy can intercept calls obtained from the container. The meter has a finite attempt-result vocabulary. It must not use an order UUID, customer, exception message, SQL statement, or configuration value as a tag. Because both increments happen before commit, reservation_requested describes a reservation attempt accepted by this code path; it is not a durable committed-order outcome. Measure committed reservations after confirmed commit or from durable-state reconciliation. In an incident, compare attempt results and inventory_conflict to pool acquisition time, transaction duration, readiness, and user latency. A rising conflict rate on one SKU can be expected scarcity; rising pool wait across all routes points toward capacity, locks, query behavior, or a dependency boundary.
A safe rollback is conditional. If a newly deployed version is correlated with the onset and the previous artifact and configuration are known good, reduce traffic or canary the previous immutable version while preserving compatible schema and message contracts. Do not roll back code across a destructive or incompatible database migration without its tested rollback/forward plan. If durable orders may exist in both versions, reconcile them through stable identifiers and states, not by deleting “suspect” rows during pressure.
Failure modes and dangerous misconceptions
“Restart first.” A restart can clear a transient deadlock or leak, but it can also erase thread evidence, interrupt in-flight orders, create a retry surge, and hide a recurring configuration or query defect. Capture minimal safe evidence and choose restart only when its expected benefit and rollback effect are understood.
“More connections fix pool wait.” Pool wait is a symptom. More connections can help only when the database has spare capacity and application demand is reasonable; under lock contention or a slow database it can add work and worsen tail latency. Investigate transaction duration, locks, query cost, and admission first.
“A timeout means no order.” A gateway or client timeout can occur after the database committed. Find the idempotency record and durable order state before instructing a retry. A correct retry returns or compares the recorded outcome rather than issuing a second reservation.
“A stack trace is the root cause.” It records one failing execution. It may be caused by a downstream timeout, a pool wait, a policy denial, or a bug. Compare timing, version, configuration, rate, and resource evidence before changing code or capacity.
“Green health means recovered.” Liveness can be green while readiness, latency, commit correctness, and user success remain poor. Recovery verification requires user experience, durable order invariants, and resource stability over a defined observation window.
Security, privacy, transaction, capacity, and cost implications
Incident access follows least privilege. A responder may need version, sanitized configuration, metrics, and controlled order lookup, but not unrestricted production data, environment dumps, or token logs. Make break-glass access auditable and time-bounded. Redact addresses, payment-adjacent identifiers, JWTs, cookies, and secrets from copied logs, tickets, and chat. A rushed incident response must not become a privacy incident.
Preserve transaction correctness while shedding load. Reject or queue new work at a documented admission boundary; do not kill database connections indiscriminately or delete pending records. For commands already accepted, rely on idempotency, durable state, and outbox recovery. If an external payment was attempted, reconcile from provider-safe identifiers and business state; a local transaction cannot prove remote settlement. Be explicit about which users may need notification and which commands can be retried safely.
Capacity recovery should be reversible. Reduce nonessential load, lower concurrency, disable an optional expensive feature flag, route away from a degraded replica, or scale only after identifying the bottleneck. Record the exact action, actor, timestamp, reason, expected signal, and rollback condition. Scaling database clients without a budget can increase cost and obscure the causal trail; aggressive retries consume the same pools the recovery needs.
Testing and production validation
Rehearse the runbook with bounded failure injection. Make a database query slow, create hot-row contention, exhaust a small test pool, introduce a remote client timeout, deploy a known-safe configuration change, and deny a support identity. For each scenario, practice collecting version/configuration evidence, reading Actuator readiness/liveness, distinguishing executor from pool wait, checking transaction and JPA/query evidence, applying a reversible mitigation, and rolling it back.
Test user recovery. For a request that times out after a simulated commit, prove that the customer can retry with the same idempotency key and receive the recorded outcome. For a request rejected before acceptance, prove that the client receives a documented retryable or nonretryable response. Test that authorization and tenant ownership still protect support diagnostics under incident mode. Test migrations and rollback plans before a release rather than during a live incident.
Validate production recovery in three dimensions. User evidence includes accepted traffic, error rate, latency, and support outcomes. Correctness evidence includes committed order state, inventory invariants, idempotency replay, outbox age, and reconciliation backlog. Resource evidence includes pool active/pending counts, lock wait, query latency, CPU, memory, GC, executor queue, and readiness stability. Observe all three after mitigation and again after rollback of the mitigation; one momentary green graph is insufficient.
Operations checklist
- Declare an incident lead, timebox actions, and keep a timestamped evidence/action log.
- Stabilize users with bounded admission, clear status, and idempotent retry guidance.
- Capture immutable version, deployment, profile, feature flag, and sanitized configuration evidence.
- Classify failures at validation, error formatting, security, service, transaction, database, thread, or remote boundaries.
- Check whether an order/idempotency/outbox record committed before asking a customer to retry.
- Inspect pool wait, transaction duration, lock wait, query/JPA evidence, and executor queues before scaling.
- Use protected Actuator diagnostics; distinguish liveness from readiness and user success.
- Apply the smallest reversible mitigation with an explicit expected effect and rollback condition.
- Do not roll back across incompatible migration or message contracts without a rehearsed plan.
- Verify user, correctness, and resource recovery over a defined stable window.
Official sources
- Spring Boot production-ready features — accessed 2026-08-05.
- Spring Boot graceful shutdown — accessed 2026-08-05.
- Spring Framework declarative transactions — accessed 2026-08-05.
- Spring Data JPA reference — accessed 2026-08-05.
Continue the learning path
Use Spring Boot Actuator, Health, and Metrics Explained to build the safe signals this runbook needs, then revisit Spring Service Layers and Transaction Boundaries Explained for durable recovery boundaries. Continue through the Spring Backend course, Topics, and Spring Backend Engineering.