Kubernetes exposes desired and observed workload state, scheduling events, container termination reasons, and resource measurements. Application telemetry explains request and business behavior. Neither layer replaces the other.
Quick answer
Start with bounded user impact and the recent change window. Compare Deployment desired, updated, ready, and available replicas; inspect Pod phase, conditions, events, restart count, last termination reason, probes, CPU throttling, memory, and scheduling. Correlate those signals with route-level metrics, protected logs, sampled traces, dependencies, and durable business records. Avoid secrets, personal data, raw tokens, and unbounded identifiers in telemetry.
Learning objectives
- Select the next Kubernetes or application signal from an observed symptom rather than collecting every diagnostic indiscriminately.
- Separate controller, scheduler, runtime, process, dependency, and business evidence while preserving safe correlation.
Prerequisites
Complete Safe Backend Releases, Structured Logging, and the course. Browse the topic cluster.
Evidence by layer
- Deployment: desired, updated, ready, available, unavailable, progressing condition, revision.
- Pod: phase, readiness, scheduling condition, node, restarts, deletion timestamp.
- Container: current waiting/running state, last termination reason, exit code, resource use.
- Traffic: Service endpoints, route, response class, latency, retry and rejection behavior.
- Application: bounded error category, trace, pool wait, queue age, dependency result.
- Business: committed order, idempotency state, outbox event, reconciliation status.
Use low-cardinality labels such as workload, namespace, route template, outcome class, and deployment digest. Keep customer IDs, request bodies, credentials, raw URLs, and error messages out of metric labels. Correlation IDs should be opaque and access-controlled, not carry personal data.
Production failure scenario
After a rollout, restart count rises but current Pods look healthy. lastState.terminated.reason shows OOMKilled on the new digest. Application heap graphs look normal because native buffers grew. Logs contain only the successful restart path.
Correlate restart timing, container memory, process native memory, traffic, and the digest. Mitigate with the approved release control, then reproduce under representative limits. Current state alone hid the previous termination evidence.
Common misconceptions
- Pod phase
Runningdoes not mean every container is Ready or the service is correct. - Events are useful but are not an unlimited audit log.
- Restart count without last termination reason is incomplete evidence.
- One trace is not population-level proof.
- Adding high-cardinality labels to metrics can expose data and damage the telemetry system.
Decision checklist
- Bound impact by route, outcome, version, region, and time without sensitive dimensions.
- Preserve the recent deployment digest and configuration revision.
- Read controller conditions, Pod events, and last container state together.
- Correlate infrastructure waits with application pools, queues, traces, and dependencies.
- Verify durable business state for ambiguous requests.
- Record what mitigation changed so later evidence is interpreted correctly.
Continue with Production Kubernetes Backend Incident Troubleshooting.
Related reading
- Kubernetes Rolling Updates and Safe Backend Releases
- Distributed Tracing with OpenTelemetry
- Production Logging Best Practices
Official sources
- Kubernetes debugging applications, accessed August 18, 2026.
- Kubernetes Pod lifecycle, accessed August 18, 2026.
- OpenTelemetry signals, accessed August 18, 2026.