Liveness, Readiness, and Startup Health Checks Explained

Quick answer

In Kubernetes, a startup probe answers whether a container has finished starting. While it has not succeeded, Kubernetes does not run that container’s liveness or readiness probes. A liveness probe answers whether restarting the container is an appropriate remedy for a local, unrecoverable loss of progress. A readiness probe answers whether the Pod should currently receive regular Service traffic. These are different control signals, not three names for one dependency check.

Keep liveness cheap, local, and conservative. It should not query the database, queue, another service, DNS, or a third party: restarting the application does not repair a remote outage and removes capacity when it is most valuable. Startup should allow the measured initialization envelope. Readiness can consider whether the instance can accept routed work, but taking every Pod out of service during shared dependency pressure can turn partial degradation into total unavailability.

Probes do not replace request admission, load shedding, concurrency limits, deadlines, bounded queues, or autoscaling. Give probe handlers reserved execution capacity and minimal responses. Monitor failures, transitions, container restarts, endpoint membership, user SLI, and resource pressure together. Tune periods, timeouts, and thresholds from startup and failure tests; example values are not universal defaults.

Shared flash-sale incident stage

A flash sale produces checkout overload and order API saturation. Order handlers fill workers and wait for inventory connections. Inventory database pool wait rises, requests retain no useful deadline budget, and callers retry. Oversized queues conceal the problem until old work and memory pressure accumulate.

The service’s /health handler runs an inventory query for all three probe types on the same executor as checkout. Under database pressure, readiness fails across replicas. Endpoint removal causes capacity loss and concentrates traffic on the remaining Pods. Liveness then sees the same timeout and kills otherwise recoverable processes. Cold replacements warm caches and connections, immediately face the remote outage, and fail again. This probe-induced restart cycle becomes a restart storm, reducing throughput and adding connection churn.

Layered mitigation stops the spiral. Freeze the rollout, suppress redundant retries, pause background consumers, shed low-priority work, and reserve resources for interactive checkout and cheap local probes. Temporarily correct an unsafe liveness configuration through the approved rollout mechanism rather than disabling all health observation blindly. Do not enlarge the inventory pool without database evidence.

Recovery requires more than green probe responses. Restore traffic gradually and inspect checkout success and latency, in-flight work, admission rejection, oldest queue age, pool wait, retry volume, readiness transitions, restart count, startup duration, and EndpointSlice membership. Keep the incident active until capacity membership is stable and warm Pods sustain normal user outcomes without another probe transition wave.

Core mechanism and evidence boundary

The behaviors in this article are Kubernetes specific. Other orchestrators may use similar words but have different routing, restart, grace-period, and threshold semantics. In Kubernetes, the kubelet executes configured probes. Failed liveness beyond its threshold causes a container restart subject to restart policy. Failed readiness marks the Pod not ready for matching Service endpoints. A startup probe delays both liveness and readiness execution until startup succeeds; repeated startup failure also leads to a restart.

Design endpoints around the action. Startup checks only local initialization required before any traffic, such as loaded validated configuration and completed in-process state setup. Liveness checks local progress conditions for which process replacement is useful, such as an unrecoverable event-loop deadlock detected by an independent watchdog. Remote dependencies must never determine liveness. Otherwise a database or queue outage can create a restart storm and destroy healthy local capacity.

Readiness answers whether this instance should receive regular routed work. It may include local admission state, draining state, and required local initialization. Treat remote checks cautiously: if every Pod shares the dependency, simultaneous removal may leave no endpoint to return an intentional degraded or overload response. Readiness is not the sole overload control; keep per-request deadlines, admission, load shedding, bounded queues, bulkheads, and safe fallback policy.

The evidence boundary uses consistent meanings. route is a low-cardinality dimension containing a bounded route template, never a raw path. operation is a low-cardinality dimension for stable named work such as probe.readiness or order.create. priority is a low-cardinality dimension for a bounded class or tier such as probe or interactive. admission is a low-cardinality dimension for an accepted or denied capacity decision. rejection is a low-cardinality dimension for the denied-work decision with bounded values such as draining.

deadline.remaining_ms is a teaching field that records milliseconds, not a stable OpenTelemetry semantic convention. Use versioned runtime and telemetry conventions in production. trace_id, request_id, and message_id remain correlation fields in protected events. Never use metric labels or metric dimensions for trace_id, request_id, message_id, order_id, user_id, a raw URL, exception text, a credential, a token, or PII. Probe dimensions should be bounded to probe type, result, workload, namespace, and similarly reviewed values.

Probe counters and Kubernetes events provide population evidence; sampled traces can explain selected handlers but are not population data. Distinguish a probe HTTP failure, a readiness state transition, EndpointSlice propagation, container termination, and completed restart. They occur at different times. Missing probe telemetry is unknown, not proof of success.

Minimal reproducible implementation

The following is a complete Kubernetes Deployment example. The image, resources, ports, and timing values are illustrative inputs that must be replaced by an immutable production image and values measured for the application, nodes, and rollout policy.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
  namespace: commerce
  labels:
    app.kubernetes.io/name: order-api
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: order-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: order-api
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: order-api
          image: registry.example.com/commerce/order-api:1.42.0
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "2"
              memory: "1Gi"
          startupProbe:
            httpGet:
              path: /health/startup
              port: http
              scheme: HTTP
            periodSeconds: 2
            timeoutSeconds: 1
            failureThreshold: 30
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /health/live
              port: http
              scheme: HTTP
            periodSeconds: 10
            timeoutSeconds: 1
            failureThreshold: 3
            successThreshold: 1
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
              scheme: HTTP
            periodSeconds: 5
            timeoutSeconds: 1
            failureThreshold: 2
            successThreshold: 2

Implement the handlers as independent, allocation-light paths:

GET /health/startup:
  if validatedConfigurationLoaded and localInitializationComplete:
    return 204
  return 503

GET /health/live:
  if watchdogConfirmsLocalProgress and not unrecoverableLocalCorruption:
    return 204
  return 503

GET /health/ready:
  if startupComplete and not draining and localAdmissionGateCanServeProbe():
    return 204
  return 503

None of these handlers performs a database query, queue operation, DNS lookup, or third-party request. Application metrics can separately report dependency health. If product policy requires readiness to account for a dependency, model the all-Pod failure behavior and preserve a route for honest degraded responses. Keep the handler response body minimal; Kubernetes HTTP probe success is determined from status code.

The startup budget in the manifest is approximately sixty seconds of failed checks after probe execution begins, but scheduling, image pulling, endpoint propagation, and termination are separate intervals. Measure actual cold and warm startup distributions and failure detection goals before selecting values.

Interpret thresholds as a policy tested against actual kubelet behavior, not as an exact wall-clock timer. failureThreshold counts consecutive failures, while probe duration, scheduling delay, node pressure, and the configured period shape the observed detection interval. Kubernetes may run readiness checks more frequently while a container is not ready. successThreshold must stay at one for startup and liveness, whereas readiness can require consecutive successes to damp a flapping route decision. Record probe-attempt and state-transition timestamps separately so an operator can distinguish a slow handler from controller and endpoint propagation.

Probe validation also includes termination. When a Pod begins graceful deletion, endpoint state and process shutdown progress concurrently; existing connections and already admitted work may still be present. The application should stop new admission on its local termination signal, allow only bounded in-flight work to finish, release permits and connections, and exit within the configured grace period. A readiness transition does not recall requests already delivered, so handlers still need deadlines, cancellation, and idempotent outcomes. Test the full interval from deletion request through EndpointSlice change, termination signal, final response, and forced-shutdown boundary rather than declaring success when the readiness endpoint first turns red.

Failure modes and dangerous misconceptions

“One deep health endpoint is simpler.” The restart decision and routing decision have different consequences. A shared deep check couples remote failure to local replacement and can remove every endpoint.

“Liveness should prove the whole service works.” Liveness asks whether restarting this container helps. It should be conservative because false failure destroys capacity and state such as warm caches and connections.

“Readiness is load shedding.” Endpoint removal is coarse, delayed, and can concentrate traffic. Per-request admission and bounded concurrency react at the work boundary and can preserve explicit outcomes and priorities.

“A longer initial delay handles slow startup.” Startup duration can vary. A startup probe separates initialization tolerance from the ongoing liveness policy and prevents liveness and readiness execution before startup succeeds.

“Fast probing gives fast recovery.” Aggressive periods and timeouts consume resources, react to transient pauses, and can synchronize across Pods. Select detection time from the failure objective and test under CPU throttling, garbage collection, and overload.

“Green means healthy capacity.” A local endpoint can respond while the application is near its concurrency ceiling or dependency pool is saturated. Pair probes with the user SLI and RED and USE evidence.

“Deleting Pods fixes saturation.” Replacement may worsen database connection churn and cold-start demand. First determine whether the fault is local and restart-remediable.

“Readiness failure immediately stops all traffic.” Endpoint updates and load balancers have propagation and connection behavior. Continue graceful draining and enforce server-side admission during transitions.

Security/privacy/capacity/cost implications

Probe endpoints should disclose minimal state, require no public secret, and avoid configuration, build internals, dependency names, stack traces, or customer data. Restrict external exposure through network and routing policy. Do not make the kubelet depend on a rotating application token that can expire independently and restart the fleet.

Reserve enough CPU, worker, and connection-independent execution for probes and termination handling. A probe that enters the normal request queue can fail because business traffic filled it. Conversely, an entirely separate thread can hide a deadlocked application; use a watchdog that confirms relevant local progress without invoking remote systems.

False restarts cost cold caches, image and network traffic, connection churn, longer recovery, and error budget. Overly permissive probes can leave a truly stuck process indefinitely. Capacity planning should model maximum unavailable Pods, rolling update surge, node loss, startup distribution, dependency limits, and probe traffic. Configuration values need version control and change review.

Testing and production validation

Unit-test each endpoint’s local state matrix. Verify liveness remains successful during database, queue, DNS, and third-party outages while dependency metrics report failure. Verify readiness changes during an intentional drain and returns only after the configured success threshold. Confirm startup blocks the later probes until initialization finishes and restarts a permanently stuck startup according to policy.

In an isolated environment, inject event-loop deadlock, CPU throttling, memory pressure, long garbage collection, dependency latency, lost replicas, and a rolling update. Observe probe request latency, failures, state transitions, container restarts, endpoint membership, traffic distribution, and the user SLI. Confirm one dependency outage does not restart every Pod.

Canary timing or endpoint changes. Compare actual detection and recovery with the objective and check for synchronized transitions. During overload, verify request admission and load shedding protect probe execution without falsely claiming business capacity. During recovery, wait for stable endpoint membership, warm capacity, normal pool wait, drained useful backlog, controlled retries, and sustained checkout success.

Test lifecycle edges that ordinary steady-state probes miss. Send termination, confirm readiness changes and new admission stops, allow in-flight work only within the grace period, and verify the process exits without accepting replacement work late. Restart the kubelet or temporarily lose probe networking and inspect actual events rather than inferring behavior from application counters alone. Exercise a bad image and invalid probe path in a disposable namespace so configuration failure is distinguishable from application overload. Check rolling updates with the declared surge and unavailable settings, plus a node drain and one unavailable zone. The surviving capacity must still meet the tested admission envelope; healthy probe configuration cannot compensate for an update policy that removes too many replicas at once.

Operations checklist

• Document the precise restart action, routing action, and startup gate for Kubernetes.
• Keep liveness local and exclude database, queue, DNS, and third-party health.
• Use startup protection for measured initialization rather than weakening ongoing liveness.
• Make readiness reflect instance routing eligibility, draining, and local admission state.
• Retain deadlines, bulkheads, bounded queues, load shedding, and fallback policy.
• Reserve lightweight execution capacity for probes and termination handling.
• Monitor probe attempts, transitions, EndpointSlice membership, and container restarts.
• Correlate probe changes with user SLI, in-flight work, queue age, pool wait, and retries.
• Test thresholds under throttling, pauses, dependency failure, rollout, and recovery.
• Restore capacity gradually and verify warm, stable service before closing an incident.

Official sources

• Kubernetes: Liveness, Readiness, and Startup Probes
• Kubernetes: Pod Lifecycle
• Kubernetes: EndpointSlices
• Google SRE: Addressing Cascading Failures

Accessed 2026-08-02. Verify behavior against the Kubernetes version and controllers actually deployed; probe thresholds remain workload-specific engineering decisions.

Inside the application, Spring Boot Actuator, Health, and Metrics Explained maps these orchestration actions to minimally exposed, consumer-specific health groups and low-cardinality signals. During an outage, Production Spring Boot Incident Troubleshooting keeps a probe transition in context with user, transaction, pool, lock, and durable-state evidence before restart or rollback.

Related reading

Place probes within Production Resilience for Backend Systems, preserve capacity with Load Shedding and Adaptive Concurrency Limits, and isolate probe execution through Bulkhead Pattern and Resource Isolation. Use Production Overload Troubleshooting for incident evidence. Continue via System Design and the Topics index.