Production Database Engineering · Lesson 16

Production Database Incident Troubleshooting

Diagnose pool exhaustion, lock contention, slow plans, failed migrations, replica lag, storage pressure, and failover ambiguity.

Quick answer

The Spring order service remains the user-impact and durable-state case throughout this incident workflow.

Use one fixed sequence: bound user impact, identify recent change, collect layered evidence, choose reversible mitigation, verify business recovery, and preserve evidence for follow-up. A restart, scale action, rollback, promotion, or green health check is mitigation evidence—not proof that orders recovered.

Learning objectives

  • Differentiate connection, transaction, lock, plan, storage, replication, migration, and failover symptoms.
  • Verify PostgreSQL 18 or MySQL 8.4 recovery with durable order and user evidence.

Prerequisites

Know the preceding course boundaries and your organization’s incident authority, privacy policy, and database runbooks.

Shared relational contract

Start with affected operation, cohort, onset, severity, and durable risk. Record recent application, schema, configuration, traffic, maintenance, and topology changes. Preserve bounded logs, metrics, traces, plans, lock evidence, and identities before destructive mitigation.

Prefer admission control, workload isolation, cancellation of a confirmed offender, compatible application rollback, or governed failover according to evidence. Every action needs a stop condition and rollback path. After mitigation, reconcile ambiguous requests and backlogs.

PostgreSQL 18 boundary

Use activity, wait events, locks, plans, table and index statistics, vacuum age, WAL, replication, logs, and recovery state with PostgreSQL meanings. A backend termination may roll back its local transaction but cannot reverse already committed or external effects.

MySQL 8.4 InnoDB boundary

Use Performance Schema threads and waits, data locks, InnoDB transaction and status evidence, plans, redo and binary log state, replication status, and error logs. A killed connection has engine-specific rollback cost and does not classify an already acknowledged business outcome.

Production failure scenario

A migration is followed by pool saturation and checkout errors. Preserve the migration identity and engine state. Determine whether sessions wait on metadata, row locks, I/O, or slow plans. Pause further rollout, bound traffic, and choose the least destructive compatible action. Verify schema, replicas, new orders, affected idempotency keys, outbox progress, and error objectives before recovery declaration.

Common misconceptions

  • Restarting clears evidence and may amplify load.
  • Scaling application replicas can multiply database sessions.
  • Rolling back code may not reverse schema or data.
  • Replica catch-up does not reconcile ambiguous customer outcomes.

Decision checklist

  • Bound impact and durable-state risk.
  • Record recent changes and exact identities.
  • Collect safe application and engine evidence.
  • Choose a reversible action with stop conditions.
  • Reconcile ambiguous and rejected orders.
  • Preserve evidence and assign corrective work.

Review observability and capacity, finish with the lab guide, use the course and topic, deepen with production slow-query troubleshooting, and connect to production incident troubleshooting and observability.

Official sources

Knowledge check

Check your understanding

Answer both questions correctly to mark this lesson as mastered. You can retry without penalty.

1. Checkout errors rise immediately after a migration; what is the first operational sequence?

2. Restarting the database clears waiting sessions; what establishes recovery?