Cloud-Native Backend Operations · Lesson 12

Kubernetes Rolling Updates and Safe Backend Releases

Coordinate immutable images, surge, unavailability, readiness, termination, rollback, and compatible database change in a backend rollout.

A safe release keeps old and new application versions compatible while capacity temporarily includes starting, ready, and terminating replicas. Kubernetes coordinates replacement; the release process owns artifact identity, data compatibility, user evidence, and stop conditions.

Quick answer

Deploy an immutable reviewed image digest. Set maxSurge and maxUnavailable from real spare capacity and availability policy, use truthful readiness, and give terminating Pods enough bounded drain time. Change databases through expand, migrate, verify, and contract phases that support mixed versions and rollback. Deployment rollout completion does not prove business metrics, migrations, jobs, or consumers succeeded.

Learning objectives

  • Derive rollout concurrency from workload capacity, cluster headroom, readiness delay, and termination behavior.
  • Keep database, event, configuration, and application contracts compatible across old, new, and rollback versions.

Prerequisites

Review Horizontal Pod Autoscaling and Zero-Downtime Database Schema Migrations. Follow the course and topic.

Rollout and data gates

maxSurge bounds extra Pods above desired replicas; maxUnavailable bounds unavailable desired capacity during the update. Percentages have controller-defined rounding semantics, so review the resolved integer behavior for small replica sets. Surge also consumes node, connection, cache, and downstream capacity.

Database change should remain compatible while versions overlap:

  1. Expand schema and contracts additively.
  2. Deploy compatible writers and readers.
  3. Run bounded, resumable migration or backfill.
  4. Reconcile correctness and prove old writers are gone.
  5. Contract obsolete fields only in a later release.

Rollback must be tested against the expanded state; dropping a column in the same rollout removes that option.

Production failure scenario

A three-replica service uses maxSurge: 100%. Six Pods can coexist, and each opens a 40-connection pool. The database limit is 160. New Pods become Ready locally, but pool waits and errors rise before old Pods terminate.

Pause or reverse through the approved release control, protect admission, and verify database recovery. Reduce pool or surge ownership and retest. The Deployment controller behaved correctly; the capacity contract was wrong.

Common misconceptions

  • Rollout completion does not prove business success, migration completeness, or consumer compatibility.
  • Rollback cannot undo an incompatible destructive schema change.
  • Surge capacity is not free cluster or dependency capacity.
  • Readiness success does not prove caches, jobs, or downstream effects are correct.
  • Reusing a mutable tag can make rollback identity ambiguous.

Decision checklist

  • Promote and record an immutable image digest.
  • Resolve integer surge and unavailable behavior for the actual replica count.
  • Budget old, new, starting, and terminating resource ownership together.
  • Test mixed-version reads, writes, jobs, events, and rollback.
  • Define automated and human stop conditions from user and dependency evidence.
  • Verify business outcomes after controller convergence before declaring success.

Continue with Kubernetes Observability and Debugging for Backend Workloads.

Official sources

Knowledge check

Check your understanding

Answer both questions correctly to mark this lesson as mastered. You can retry without penalty.

1. A rollout reaches Available replicas, but old consumers fail on rows written by the new version; what release conclusion is valid?

2. The new release requires dropping a column used by old Pods; which sequence supports rollback?