Production Alerting, On-Call, and Runbooks

Quick answer

Page an operator when users are suffering, the condition is urgent, and a safe action exists. Create a ticket for capacity, hygiene, or trend work that can wait. A production alert needs a user-facing symptom, severity, owner, runbook, evidence links, deduplication policy, escalation path, and explicit recovery criteria.

For the Spring order service, a fast checkout error-budget burn can page. Database pool utilization, Collector queue growth, or one instance’s CPU are diagnostic signals unless they independently threaten a user objective and have an actionable response. Alert acknowledgment starts human ownership; it does not prove mitigation or recovery.

Learning objectives

• Route symptom-based SLO alerts to page or ticket actions with ownership, deduplication, inhibition, and escalation.
• Write runbooks and handoffs that preserve user impact, evidence quality, reversible mitigations, and explicit recovery criteria.

Prerequisites

Understand SLIs, SLOs, Error Budgets, and Burn-Rate Alerts, telemetry freshness, and basic incident severity.

Evidence and system boundary

Prometheus evaluates alerting rules and sends alert instances to Alertmanager. Alertmanager groups, deduplicates, silences, inhibits, and routes notifications. A pager or chat system delivers a notification. None of these components verifies that a human understood the incident, a mitigation worked, or the order service recovered.

Alert on symptoms as high in the user journey as trustworthy evidence allows. Attach component evidence for diagnosis. Do not encode every possible cause as a page, and do not silence a symptom merely because one suspected cause is quiet.

alert_contract:
  name: CheckoutFastBurn
  owner: order-on-call
  user_symptom: checkout-good-event ratio
  urgency: page
  runbook: /runbooks/checkout-fast-burn
  recovery: fresh SLI, telemetry freshness, dependency capacity, durable outcomes

This example is documentation, not a live alert configuration or proof of a production response.

Runbooks and on-call flow

A runbook starts with scope and safety: what users see, which services and regions participate, what evidence may be missing, and which actions require approval. It lists the safest diagnostic pivots and reversible mitigations before destructive or capacity-expanding actions. It specifies exit criteria and how to handle ambiguous orders.

Group notifications that share user impact and time. Inhibit redundant component pages when one incident already owns the symptom, while retaining component data in dashboards. Silences require an owner, reason, start, expiry, and review; an indefinite silence converts a known risk into invisible risk.

Handoffs include current user impact, timeline, hypotheses with supporting and contradicting evidence, changes already made, pending approvals, next checkpoint, and explicit command ownership. The outgoing responder remains responsible until the incoming responder acknowledges the handoff.

Production failure scenario

Checkout burn triggers one page while inventory latency, database pool saturation, retry growth, and Collector export failures produce five more. Responders split across alerts and apply conflicting changes. One alert clears after a rollback, but telemetry is still delayed.

The incident commander keeps the checkout page primary, groups the related evidence, inhibits duplicate pages, and assigns one operations owner. The runbook requires fresh synthetic and SLI evidence, Collector recovery, pool wait, retry volume, and durable order reconciliation before closure.

Common misconceptions

• More pages do not guarantee earlier detection; they can reduce attention and ownership.
• A silence is not a fix and must expire.
• Alert acknowledgment is not mitigation.
• A successful rollback command is not user recovery.
• Resource thresholds are not automatically user-impact objectives.
• Runbooks should constrain unsafe action, not merely collect dashboard links.

Decision checklist

• Tie urgency to user impact and an available action.
• Give every alert one accountable owner.
• Separate page, ticket, and informational events.
• Group and inhibit related symptoms deliberately.
• Require bounded silences with expiry and reason.
• Link current evidence and safe first actions.
• Preserve ambiguous durable outcomes.
• Define recovery across users, telemetry, dependencies, and correctness.
• Rehearse escalation and handoff paths.

Related reading

Previous: SLIs, SLOs, Error Budgets, and Burn-Rate Alerts. Next: Incident Command, Communication, and Postmortems. Follow Production Observability & SRE and its topic.

Connect alert actions to Production Resilience for Backend Systems and Production Kubernetes Backend Incident Troubleshooting.

Official sources

• Prometheus alerting practices
• Alertmanager overview
• Google SRE: Practical Alerting
• Google SRE: Being On-Call

Official sources accessed August 19, 2026. Example thresholds and routing policies must be validated for the owning service rather than copied as universal defaults.