Quick answer
Incident management separates command from technical execution and stakeholder communication. One incident commander owns priorities and coordination; an operations lead owns technical work; a communications lead publishes approved facts; a scribe maintains the timeline and decisions. Roles can be combined in a small incident, but ownership must remain explicit.
A postmortem is a factual learning artifact, not a verdict on one person. Record user impact, detection, timeline, evidence gaps, contributing conditions, mitigation, recovery proof, and follow-up actions with owners and due criteria. Preserve uncertainty instead of inventing a single root cause.
Learning objectives
- Separate incident command, operations, communications, and record-keeping responsibilities while maintaining one factual state document.
- Produce a blameless postmortem with impact, timeline, contributing conditions, evidence limits, and owned follow-up work.
Prerequisites
Know the alert and runbook flow from Production Alerting, On-Call, and Runbooks and basic reversible incident mitigation.
Evidence and system boundary
The incident process coordinates decisions; it does not replace technical evidence. Chat messages, tickets, commands, dashboards, and verbal reports have different reliability and retention. The live state document records what is known, what remains a hypothesis, who owns each action, and when the next update is due.
Communication must use bounded approved facts. Do not paste credentials, personal data, customer payloads, raw logs, or unrestricted trace baggage into broad channels. Security, legal, privacy, and customer communication escalation follow named policies outside the technical channel.
Impact: checkout requests are slow; durable outcome for timed-out orders is under review.
Known change: order-service version changed at the recorded deployment time.
Hypothesis: inventory pool contention contributes to latency; not yet proven.
Action owner: operations lead evaluates reversible rollback.
Next update: after fresh SLI, telemetry freshness, and durable-state checks.This is an educational state-document fragment, not a report of a real incident.
Command and communication flow
Declare severity from user harm, correctness risk, scope, and urgency. Establish one command post and live document. Freeze uncoordinated production changes. Assign investigations with explicit questions instead of asking everyone to search the same logs.
The incident commander chooses priorities and resolves conflicts but need not be the deepest technical expert. Operations proposes bounded changes and records expected outcomes and rollback. Communications separates internal technical detail from customer-safe facts. The scribe timestamps alerts, decisions, actions, observations, and handoffs.
Recovery is a phase, not an instant. Continue until fresh user requests succeed, telemetry is trustworthy, constrained dependencies stabilize, queues drain within policy, and ambiguous order or payment outcomes have an owner. State clearly what has not been proven.
Postmortem actions should change a system, guardrail, test, runbook, ownership boundary, or evidence path. “Be more careful” is not testable. Each action needs an owner, priority, completion evidence, and review of whether it reduces recurrence or detection/recovery risk.
Production failure scenario
A Spring order service deployment coincides with rising checkout latency. Several responders independently roll replicas, change pool limits, and silence alerts. Stakeholders receive conflicting recovery messages. The missing timeline makes it impossible to separate cause, mitigation, and observation.
The team establishes command, stops unowned changes, records the version and evidence gaps, assigns one rollback, and publishes a factual update. The postmortem later identifies weak change ownership, alert fan-out, incomplete Collector monitoring, and missing order-reconciliation steps as contributing conditions—not individual carelessness.
Common misconceptions
- The incident commander is not automatically the person typing commands.
- A busy chat channel is not a maintained incident state document.
- Temporal proximity to a deployment is not proof of causation.
- Blameless does not mean actionless or consequence-free.
- One “root cause” label can hide multiple necessary conditions.
- Closing the incident does not close every investigation or follow-up action.
Decision checklist
- Assign command, operations, communications, and scribe ownership.
- Maintain one timestamped source of current facts.
- Label hypotheses and contradictory evidence.
- Record every production change, owner, expectation, and result.
- Use approved channels without leaking sensitive evidence.
- Require explicit handoffs.
- Verify user, telemetry, dependency, and durable-state recovery.
- Write blameless contributing conditions.
- Assign measurable follow-up work with owners.
Related reading
Previous: Production Alerting, On-Call, and Runbooks. Next: Production Incident Troubleshooting with Logs, Metrics, and Traces. Follow Production Observability & SRE and its topic cluster.
Use Production Database Incident Troubleshooting for durable-state incidents and Saga Pattern and Compensating Transactions for long-running recovery ownership.
Official sources
- Google SRE: Managing Incidents
- Google SRE: Emergency Response
- Google SRE: Postmortem Culture
- Google SRE Workbook: Postmortem Culture
Official sources accessed August 19, 2026. Organization-specific severity, disclosure, privacy, and escalation policies remain authoritative for real incidents.