SLI, SLO, Error Budgets, and Burn-Rate Alerts Explained

Quick answer

A service level indicator (SLI) measures a user-relevant outcome. A service level objective (SLO) is a target for that indicator over a window. The error budget is the allowed fraction of bad events: a 99.9 percent availability objective permits 0.1 percent bad events. Burn rate compares the observed bad-event rate with that allowed rate. A burn rate of 10 consumes budget ten times faster than planned.

Good SLO alerts page when user impact is significant and budget is being consumed quickly. Multi-window burn-rate alerting combines a shorter window that detects active failure with a longer window that confirms sustained impact. It is more actionable than paging on one instantaneous latency or CPU threshold. Resource alerts still help diagnosis and capacity planning, but urgency should follow user consequences.

Define the measurement boundary carefully. Decide which requests are valid, which outcomes are good, where the measurement is taken, how retries and canceled requests count, and what happens during missing telemetry. A sampled trace set is not an acceptable denominator for a traffic SLI. Use complete counters or another audited event source.

Learning objectives

• Define event-based SLIs and SLO windows for the Spring order service from user outcomes rather than component convenience.
• Use error budgets and multi-window burn-rate alerts while preserving telemetry quality and service-specific threshold assumptions.

Prerequisites

Understand OpenTelemetry Collector Pipeline, request counters, latency distributions, and basic percentages.

Production failure scenario

The shared incident begins when a slow query in the inventory service exhausts its database connection pool.

An inventory query becomes slow and exhausts the service connection pool. The order API waits and retries, increasing traffic. Reservation messages form a queue backlog. Many checkouts still complete, but too slowly, so the latency SLO burns before the raw error percentage becomes alarming.

The primary SLI counts eligible checkout attempts completed successfully within 750 milliseconds. The numerator counts good events; the denominator counts valid attempts at a stable boundary. A separate availability SLI counts eligible successful outcomes regardless of latency. Queue processing has its own freshness SLI based on reservation completion within a defined time, rather than pretending the HTTP SLO covers delayed asynchronous work.

Metrics use low-cardinality route, status, and operation dimensions with service.name and deployment.environment.name. Investigation can pivot to logs and traces using trace_id, span_id, request_id, and message_id, but those values never enter the SLI’s metric labels. During mitigation, operators roll back the query change. Recovery requires the short and long burn windows, pool wait, retry volume, and queue age to improve—not merely one successful request.

Evidence and system boundary

An SLI should represent the experience of a population of events. Request-based SLIs often use good events divided by valid events. Time-based availability can hide frequent short failures and traffic variation, so event-based measures are usually clearer when requests are observable. Window-based SLIs can model batch or synthetic outcomes but require explicit window semantics.

The SLO window may be rolling or calendar-aligned. A 30-day rolling window supports continuous operational decisions; a calendar month may match reporting. The choice affects burn calculations and reset behavior. Document exclusions such as clearly invalid client requests, but do not exclude server failures simply because they make the number worse.

Error budget is a policy input, not permission to ignore users. It helps balance reliability and change. A team may slow risky releases when budget is exhausted, prioritize reliability work, or require stronger review. The policy should name decision owners and exceptions.

Burn-rate alert thresholds are derived from objective, window, desired detection time, and budget fraction. Published Google SRE examples use multiple windows and rates; copy the method, not numbers without recalculation. Low-traffic services need special handling because one event can dominate a ratio and statistical confidence is weak.

Minimal implementation

Suppose checkout_attempts_total labels outcomes as good or bad after applying the latency and correctness contract. A 30-day SLO target is 99.9 percent:

SLO = 0.999
allowed bad fraction = 1 - SLO = 0.001
burn rate = observed bad fraction / 0.001

PromQL for a five-minute bad-event ratio:

sum(rate(checkout_attempts_total{environment="production",outcome="bad"}[5m]))
/
sum(rate(checkout_attempts_total{environment="production"}[5m]))

Convert it to burn rate:

(
  sum(rate(checkout_attempts_total{environment="production",outcome="bad"}[5m]))
  /
  sum(rate(checkout_attempts_total{environment="production"}[5m]))
)
/
0.001

A multi-window page condition requires both a fast and confirming window, for example a high burn over 5 minutes and 1 hour. A slower ticket condition may combine 30-minute and 6-hour windows at lower burn. Exact rates must be calculated from the budget fraction the organization intends each alert to consume.

Record rules reduce repeated query cost and centralize definitions:

groups:
  - name: checkout-slo
    rules:
      - record: service:checkout_bad_ratio:rate5m
        expr: |
          sum(rate(checkout_attempts_total{outcome="bad"}[5m]))
          /
          sum(rate(checkout_attempts_total[5m]))
      - record: service:checkout_burn_rate:rate5m
        expr: service:checkout_bad_ratio:rate5m / 0.001

Protect against an empty denominator. Alerting and dashboards should distinguish no valid traffic from zero bad events. Use minimum-volume conditions, synthetic traffic, or longer windows for low-traffic services. Do not silently coerce missing data into perfect availability.

Latency and availability objectives may need separate budgets. A request that succeeds after five seconds is good for availability but bad for the latency SLI. Combining every failure mode into one opaque outcome makes mitigation harder. Keep a small set of objectives tied to distinct user promises, and present them together for the journey. Avoid dozens of SLOs that operators cannot prioritize.

Asynchronous workflows need freshness rather than request latency. For inventory reservations, define the valid population of accepted messages and count those reaching a durable terminal state within the objective. Decide how dead-lettered, canceled, duplicate, and permanently invalid messages count. Queue depth is diagnostic; it is not the user promise because the same depth can be healthy at different processing rates.

Error-budget policy converts measurement into action. State what happens when remaining budget crosses thresholds, who can approve exceptions, and how planned high-risk work is evaluated. Do not stop every deployment automatically after one noisy measurement. Require trustworthy data, user impact, and a documented decision. Conversely, do not redefine exclusions during an incident to make the budget appear healthy.

Reporting should show target, current compliance, budget remaining, burn by time window, major incidents, and measurement changes. Annotate SLI definition migrations because a sudden improvement may come from a new denominator rather than a more reliable service. Review objectives periodically against user expectations and business criticality; an easy target that never influences decisions provides little value.

Failure modes and trade-offs

The wrong denominator is the most dangerous error. Counting only responses emitted by the final handler may exclude timeouts at a gateway. Counting every client validation failure may blame the service for invalid requests. Define the user journey and measure at a boundary that observes eligible attempts and their final outcome.

Retries can inflate both numerator and denominator. If one user action creates three attempts, an attempt-based SLI and a journey-based SLI answer different questions. Record both when operationally useful, but use one explicit contract for the SLO. The shared incident needs retry volume as diagnosis while checkout journey success remains the user objective.

Percentiles are not automatically SLIs. An aggregate p95 can hide a small region or route, and averaging percentiles across instances is invalid. Good-event latency thresholds are often easier to aggregate: count requests at or below 750 milliseconds over all valid requests.

Single-window alerts trade speed against noise. A short window detects quickly but flaps. A long window confirms impact but reacts late. Requiring both windows captures current and sustained burn. Still apply minimum event volume and route alerts to the team able to mitigate.

Low traffic creates uncertainty. One failure may produce a huge burn rate without broad impact. Options include synthetic checks, combining statistically similar services, using longer windows, or accepting a lower operational target. Document the choice rather than hiding it with an arbitrary suppression.

Common misconceptions

• A target is not a measurement and a dashboard is not an objective.
• Successful HTTP status can still violate a latency or correctness SLI.
• Burn-rate examples are not universal alert thresholds.
• Missing or delayed telemetry can bias the numerator, denominator, and alert window.
• Error-budget policy does not authorize unsafe changes automatically.

Security, privacy, and cost

SLI metrics should not contain personal identifiers. Use bounded route, region, tier, or operation dimensions only when each supports an operational decision and has a known maximum. Do not label by request_id, message_id, trace_id, user ID, order ID, raw URL, or exception text.

Dashboards and alert notifications can leak tenant or incident details. Restrict production views, sanitize annotations, and avoid embedding tokens in links. Error-budget policy and alert changes should be reviewed and auditable because manipulating exclusions can hide impact.

Recording rules consume series and compute resources. Precompute only combinations used by dashboards or alerts. Retain enough history to evaluate the SLO window and changes, but downsampling must preserve the good/valid calculation. Do not use sampled traces as a cheaper replacement for complete SLI counters.

Testing and validation

Build table-driven cases for good, bad, and excluded events. Include success under threshold, slow success, server error, gateway timeout, canceled request, invalid client request, and retry. Assert the exact numerator and denominator contribution. Test boundary equality at 750 milliseconds and clock or unit conversions.

Replay synthetic counters through recording and alert rules. Verify high burn triggers only when both fast and confirming windows exceed thresholds. Verify a brief spike clears without a page, sustained moderate burn creates the intended ticket, and missing data does not appear healthy. Test low traffic explicitly.

Compare the SLI with independent synthetic checks and incident records. Large disagreement suggests the measurement boundary is wrong or telemetry is missing. Monitor rule evaluation and source-series freshness.

During the shared incident, confirm the latency SLO pages while pool and queue signals diagnose. After rollback, require fresh good events, falling burn on both windows, normal pool wait, reduced retry rate, and draining queue age before closure.

Version the SLI specification beside its recording and alert rules. A review should trace each prose decision—valid event, good event, threshold, window, and exclusion—to executable logic. Require a migration note when semantics change, including how dashboards handle the boundary and whether historical compliance remains comparable.

Test partial regional failures. A global aggregate can remain within objective while one region burns rapidly. Add bounded region or tier slices for diagnosis and, where the user promise requires it, regional objectives. Do not create a separate page for every small slice; routing should reflect blast radius and ownership.

Exercise error-budget policy with tabletop scenarios. Ask whether a fast burn during a critical launch pages, who can pause the rollout, how an exception is documented, and what evidence allows resumption. Policy that has never been rehearsed will be renegotiated under pressure.

Verify asynchronous freshness after the HTTP path recovers. Queue backlog can continue violating reservation or notification objectives after checkout latency improves. Keep the incident open or transition ownership until the freshness SLO and oldest-message age prove delayed work is caught up.

Use separate alert metadata for urgency and diagnosis. The page should state the journey, objective, observed burn, affected scope, and runbook. It may link to pool or queue dashboards, but it should not claim the component cause before investigation. A ticket can include slower trends and budget projections. Review alert text in a drill so on-call responders understand the denominator, windows, and immediate safe actions without reverse-engineering the PromQL under pressure.

Keep alert-rule evaluation observable. Record evaluation failures, execution duration, missing inputs, and notification delivery. An SLO page cannot protect users if its rule is stale or its notification path is broken; test both during regular drills.

Backtest candidate windows against historical incidents and known healthy peaks, but do not optimize only for the past. Record which events would have paged, how long detection took, and how much budget had burned by then. False negatives expose unsafe thresholds; repeated non-actionable pages expose a policy or signal problem. Revisit the test after traffic shape, retry behavior, or the user journey changes, because an old calibration can become misleading without any alert-rule syntax change.

Publish the SLO definition where product, development, and operations teams can review it. Shared interpretation matters: a mathematically correct objective that stakeholders understand differently will produce disputed incident and release decisions.

Review that agreement after significant product changes.

When burn confirms urgent user impact, Production Resilience for Backend Systems maps the objective to containment and degradation controls, while Production Overload Troubleshooting requires fresh SLO, resource, backlog, and telemetry evidence before closure.

Decision checklist

• Define the user journey and valid-event population in writing.
• Prefer good events divided by valid events for request SLIs.
• Separate HTTP success, latency, and asynchronous freshness objectives.
• Document retry, timeout, cancellation, and exclusion semantics.
• Derive burn thresholds from the actual target and window.
• Pair fast and confirming windows for pages.
• Use tickets or dashboards for slower risk and capacity work.
• Handle low traffic and empty denominators explicitly.
• Keep SLI dimensions bounded and free of personal identity.
• Link every alert to an owner, runbook, dashboard, and mitigation.

Official sources

• Google SRE Workbook: Implementing SLOs
• Google SRE Workbook: Alerting on SLOs
• Google SRE: Service Level Objectives
• Prometheus recording rules
• Prometheus alerting rules

Official sources accessed August 19, 2026. Recalculate thresholds for the chosen SLO rather than copying example burn rates without their assumptions.

Related reading

Previous: OpenTelemetry Collector Pipeline. Next: Production Alerting, On-Call, and Runbooks. Follow Production Observability & SRE and its topic.

Use RED, USE, and Golden Signals to build the underlying metrics and Production Incident Troubleshooting to act on the page. The complete path is listed in System Design and Topics.