SLO Incident Triage Lab: Evidence Gaps, Alerts, and Recovery

Quick answer

The SLO Incident Triage Lab is a deterministic browser model for comparing modeled user bad events with the subset represented by modeled telemetry. It shows how capacity rejection, errors, slow requests, coverage gaps, trace sampling, short and long burn windows, mitigation, and recovery interact in teaching units.

The lab does not connect to a Spring order service, telemetry backend, Prometheus, Collector, database, Kubernetes cluster, or cloud account. It is not a benchmark, capacity calculator, SLO compliance audit, incident replay, alerting dry-run, MTTA/MTTR calculator, or production verification.

Learning objectives

• Compare modeled user-impact events with observed evidence, burn-rate windows, trace sampling, and alert state across six deterministic scenarios.
• Choose the next metrics, traces, logs, change, and dependency evidence without treating the browser model as production measurement.

Prerequisites

Complete SLIs, SLOs, Error Budgets, and Burn-Rate Alerts and Production Incident Troubleshooting.

Evidence and system boundary

Each step creates a fixed number of offered requests. Modeled capacity admits some and rejects the rest. Admitted requests can fail or complete slowly. Failed, rejected, and slow events count as bad events for the combined teaching SLI. Percentages use deterministic arithmetic, not random sampling.

Telemetry coverage produces an observed subset. Trace sampling applies only to observed requests. Alerts evaluate observed bad events, while the summary retains actual modeled impact separately. This demonstrates an evidence boundary; it does not estimate a real telemetry system’s loss or bias.

Six scenarios

1. Healthy bounded service keeps traffic within capacity and the evidence path complete.
2. Latency burns the SLO shows successful HTTP responses that are still bad user events.
3. Deployment error regression correlates a change and errors without claiming causation.
4. Capacity saturation and rejection makes admission failure visible as user impact.
5. Telemetry gap hides impact separates actual modeled bad events from observed evidence.
6. Mitigation and windowed recovery stops new incident events while long windows retain prior burn.

Open the SLO Incident Triage Lab, select a preset, inspect the timeline, change one bounded input, and run again. Identical inputs always return identical output.

Production failure scenario

Checkout latency and errors rise after a Spring order service deployment. The observed dashboard shows a smaller increase because Collector export evidence is incomplete. A rollback returns new requests to baseline, but the long-window alert remains elevated and timed-out orders may have committed.

The lab can display that arithmetic. In production, responders would still need application emission, Collector receive and export signals, SLI source quality, representative traces, protected logs, deployment evidence, dependency capacity, and durable order reconciliation.

Reading the result

Actual burn uses every modeled bad event. Observed burn uses only the modeled telemetry subset. Their difference is a teaching detection gap. Watch, firing, and recovering are model states, not Alertmanager output. Recovery step appears only after mitigation and a clean enough modeled long window; it is not elapsed time or MTTR.

The evidence cards deliberately request independent signals. Follow them before choosing a mitigation. A successful command, cleared alert, lower instantaneous burn, or one clean trace is insufficient by itself.

Common misconceptions

• The model does not forecast a production error budget.
• Telemetry coverage is not the same as trace sampling.
• Slow successful responses can be bad SLI events.
• A recent change narrows investigation but does not prove cause.
• Alert-window recovery does not reconcile ambiguous durable orders.
• More capacity in the model is not a production scaling recommendation.

Decision checklist

• State the affected user journey and good-event definition.
• Compare actual modeled and observed modeled evidence.
• Check telemetry freshness before interpreting absence.
• Use aggregate metrics to bound impact and traces/logs for selected detail.
• Correlate recent changes without assuming causation.
• Choose a bounded reversible mitigation.
• Continue through fresh short and long windows.
• Verify dependency and durable-order outcomes separately.

Related reading

Previous: Production Incident Troubleshooting. The full sequence is Production Observability & SRE and the Observability and SRE topic.

Compare the same recovery boundary with Kubernetes Rollout Capacity Lab Examples and Database Connection Contention and Recovery Lab Examples.

Official sources

• Google SRE Workbook: Alerting on SLOs
• Prometheus alerting practices
• OpenTelemetry observability primer
• Google SRE: Effective Troubleshooting

Official sources accessed August 19, 2026. The browser model intentionally does not reproduce any vendor’s alert evaluator or production telemetry pipeline.