Quick answer
RED, USE, and the four Golden Signals are complementary ways to choose operational metrics. RED measures the Rate, Errors, and Duration of request-driven services. USE measures Utilization, Saturation, and Errors for resources. The Golden Signals—latency, traffic, errors, and saturation—connect service symptoms with capacity pressure. None is a complete dashboard template; each is a prompt for evidence that answers a decision.
For the checkout path, RED shows rising inventory duration while rate is stable and errors initially remain low. USE shows the database connection pool approaching full utilization, growing waiters, and timeout errors. Golden Signals keep attention on user latency and traffic while confirming saturation. Together they prevent a common mistake: paging only when CPU crosses an arbitrary threshold.
Use counters for cumulative events, gauges for current state, and histograms for distributions such as latency. Keep labels like service.name, deployment.environment.name, route, status, and operation bounded. Never label metrics with trace_id, request_id, message_id, user ID, order ID, exception text, or raw URL. Use exemplars or backend links to pivot from aggregate metrics to representative traces.
Learning objectives
- Apply RED, USE, and Golden Signals to separate Spring order service user symptoms from constrained resource evidence.
- Build aggregatable counters, gauges, and distributions with explicit units, bounded labels, and version-qualified histogram behavior.
Prerequisites
Know Structured Logging and Correlation IDs, basic Prometheus concepts, and the difference between a request symptom and a resource cause.
Production failure scenario
The shared incident begins when a slow query in the inventory service exhausts its database connection pool.
A new inventory query becomes slow and holds database connections. The inventory service connection pool reaches its limit. The order API times out and retries, increasing request traffic. Reservation messages wait longer, so queue backlog and oldest-message age grow. Checkout latency consumes the SLO before the overall error percentage looks dramatic.
Start with a user-centric symptom: successful checkouts within the latency objective. RED for the order and inventory routes shows request rate, classified errors, and duration histograms. USE for the inventory pool shows active connections divided by configured capacity, queued borrowers as saturation, and acquisition timeouts as errors. Queue metrics show incoming rate, completion rate, depth, and age. A deployment annotation provides the change boundary.
The shared telemetry contract still matters. Resource identity uses service.name and deployment.environment.name. Logs and traces can carry trace_id, span_id, request_id, and message_id, but those high-cardinality values do not belong in metric labels. Route templates, outcome status, and normalized operation values support aggregation. If a dashboard displays raw /orders/82491, instrumentation is already leaking identity and creating unnecessary series.
Evidence and system boundary
Prometheus counters increase until process reset. Queries normally apply rate() over a window rather than graphing raw counter values. Gauges can rise and fall and represent current state. Histograms count observations in cumulative buckets and expose count and sum; they allow server-side aggregation and quantile estimation. Summaries calculate client-side quantiles and are generally harder to aggregate across instances.
Metric names should use one base unit and describe one quantity. Duration is conventionally seconds, not a mixture of milliseconds and seconds. Labels identify bounded dimensions. Every unique label set creates a time series with memory, storage, network, and query cost. Prometheus guidance recommends investigating alternate approaches when cardinality can grow large.
Metrics are aggregate evidence. A histogram quantile is an estimate shaped by bucket boundaries. A rate depends on its query window. Missing series can mean zero events, failed collection, or nonexistent label combinations. Sampling is unusual for core counters but may affect metrics derived from traces. Document origin before using a trace-derived latency metric as the SLI.
Minimal implementation
An HTTP service can expose a request counter and duration histogram with bounded labels:
http_server_requests_total{service="inventory-service",route="/inventory/reservations",method="POST",status="2xx"}
http_server_request_duration_seconds_bucket{service="inventory-service",route="/inventory/reservations",le="0.5"}
db_client_connection_pool_usage{service="inventory-service",state="used"}
db_client_connection_pool_limit{service="inventory-service"}
db_client_connection_pool_pending_requests{service="inventory-service"}
reservation_queue_oldest_message_age_seconds{operation="reserve_inventory"}RED queries:
sum by (service, route) (
rate(http_server_requests_total{environment="production"}[5m])
)
sum by (service, route) (
rate(http_server_requests_total{environment="production",status=~"5.."}[5m])
)
/
sum by (service, route) (
rate(http_server_requests_total{environment="production"}[5m])
)
histogram_quantile(
0.95,
sum by (service, route, le) (
rate(http_server_request_duration_seconds_bucket{environment="production"}[5m])
)
)USE queries for the connection pool:
sum by (service) (db_client_connection_pool_usage{state="used"})
/
sum by (service) (db_client_connection_pool_limit)
max by (service) (db_client_connection_pool_pending_requests)
sum by (service) (
rate(db_client_connection_pool_timeouts_total[5m])
)Adapt metric names to the stable conventions and libraries actually deployed. Do not invent a dashboard query before verifying exported labels. Add recording rules for expensive repeated aggregations and evaluate them at a suitable interval.
An exemplar can attach a representative trace identity to a histogram observation without turning each trace into a time series. The metrics backend and instrumentation must support exemplars. Exemplar availability is not guaranteed and must not be required for the alert itself.
Choose latency buckets from objectives and observed distributions. If checkout must complete within 750 milliseconds, include a bucket at that threshold so the good-event ratio can be calculated directly. Add a few buckets below it for normal performance and above it for diagnosis. Buckets that are all below normal latency or far above the failure range cannot answer operational questions. Review buckets when workloads change; changing them creates a new metric schema and may affect long-window comparisons.
Online services, offline workers, and batch jobs need different interpretations. RED fits an API because each request has a rate, outcome, and duration. For a consumer, rate means messages completed, errors need a disposition policy, and duration may include processing but not necessarily queue wait. Add oldest-message age or end-to-end freshness. For a batch job, record last success timestamp, duration, processed records, and terminal status rather than pretending it is a continuously serving endpoint.
Saturation must match the constrained resource. CPU run queue, thread-pool pending work, connection-pool waiters, broker lag, disk queue depth, and rate-limit rejections represent different queues. A utilization ratio alone does not reveal whether demand is waiting. Conversely, one pending item may be normal. Pair utilization with wait time, queue age, and errors, and calibrate thresholds using load tests and historical healthy peaks.
Dashboard ordering should mirror investigation. Put the user SLI and traffic at the top, then RED by service and route, then USE for likely resources, then deploy annotations and pipeline freshness. Avoid a wall of every metric. Each panel should have a question in its title, a documented unit, a sensible zero or no-data state, and a link to the recording rule or runbook. This reduces interpretation time during an incident.
Failure modes and trade-offs
Average latency hides tail pain. Ten fast requests and one extremely slow checkout can produce an acceptable average while violating user expectations. Use histogram buckets aligned with meaningful objectives and inspect percentiles or good-event ratios. Avoid excessive buckets; each bucket is another series.
Error classification can lie. Counting only HTTP 500 misses timeouts reported as 504, successful responses with invalid business results, or canceled requests. Define “good” and “bad” from the user journey. Keep status labels bounded, for example status classes or a reviewed error class rather than raw messages.
Utilization without saturation is incomplete. A pool at 100 percent may be healthy when requests never wait. A pool at 70 percent may still suffer long waits because connections are unevenly held. Track both used capacity and pending acquisition or wait duration. Queue depth without age is also ambiguous when messages are large or processing rates change.
High cardinality is a production failure mode. A route label derived from raw paths, a user identifier, or a trace ID can create millions of series. Aggregation becomes slow exactly when operators need it. Drop or transform dangerous labels at instrumentation and Collector boundaries; dashboards are too late.
One-minute alerts on noisy ratios create fatigue. A low-traffic service can move from zero to 100 percent errors after one request. Use minimum-event conditions, longer confirmation windows, synthetic checks, and SLO burn-rate policies. Resource signals diagnose; user-impact signals decide urgency.
Common misconceptions
- Average latency does not describe a latency distribution.
- CPU utilization alone does not prove or disprove user impact.
- Queue depth is not the same as queue age or caller latency.
- Native histogram support is version-dependent and not a universal cross-backend contract.
- A raw order, user, trace, or URL value is not a safe metric label.
Security, privacy, and cost
Metric systems are not anonymous databases. Labels are copied into indexes, alerts, dashboard URLs, and notifications. Do not include personal identifiers, tokens, tenant secrets, raw URLs, SQL text, or exception messages. Restrict access to production metrics and protect remote-write credentials.
Cardinality is the primary cost control. Estimate series as the product of label value counts. Ten services, twenty routes, five status values, and two environments already create two thousand combinations before instances and histogram buckets. Review new labels with concrete maximums and owner-approved queries.
Retention and resolution affect both cost and diagnosis. Long-term aggregates may support capacity planning, while high-resolution data may be short-lived. Recording rules can reduce query expense but also hide dimensions removed by aggregation. Preserve enough raw evidence for the incident window and document when downsampling changes interpretation.
Testing and validation
Generate deterministic traffic: successful requests, classified errors, known latency distributions, pool saturation, and queue delay. Compare expected event counts with exported counters. Confirm counter resets do not create negative rates. Verify histogram bucket monotonicity and that chosen buckets can express the SLO threshold.
Add a contract that rejects forbidden labels such as user_id, order_id, raw path, exception message, and trace_id. Enumerate all values for route, status, and operation in test traffic and confirm bounded growth. Query the metrics backend for total series before and after deployment.
Test dashboards with no traffic, partial scrape failure, and delayed remote write. A missing panel must not appear as a healthy zero. Confirm alerts include service, environment, user impact, runbook, and a useful dashboard link without leaking sensitive labels.
Rehearse the shared incident. The SLO signal should show user impact. RED should isolate inventory latency. USE should show pool saturation and timeouts. Queue age should show delayed asynchronous work. After rollback, rate and retry volume normalize, pool wait falls, queue age drains, and the burn rate returns below the alert threshold.
Validate aggregation across instances and deploy versions. Summing rates is usually meaningful; averaging already-calculated instance percentiles is not. Confirm a rolling deployment does not double-count requests through proxy and application instrumentation. Compare server-side and client-side latency to expose network or queue time, but name them separately.
Test alert behavior under controlled cardinality growth. Introduce many raw paths in a staging workload and prove relabeling or route normalization prevents new series. Set a cardinality budget and observe the backend signal that reports active series or ingestion rejection. A contract that only checks metric names will not catch an unbounded label value source.
Review recording rules with unit-aware test data. A duration exported in milliseconds but named seconds can move a percentile by three orders of magnitude while every query remains syntactically valid. Golden fixtures should include known bucket counts, counter resets, missing scrapes, and multiple instances so the expected PromQL result is explicit.
Verify attribute translation at the Prometheus boundary. The source resource attributes remain service.name and deployment.environment.name, while a particular exporter or backend may expose normalized labels such as service_name or deployment_environment_name. The example queries in this article use the names configured by their deployment. Record that mapping and test it after upgrades; never silently invent a second environment label or assume every OpenTelemetry-to-Prometheus path applies the same normalization.
Capacity planning uses the same evidence at a longer horizon. Compare peak healthy utilization, saturation onset, queue growth, and SLO performance rather than extrapolating CPU alone. Model how retries change offered load and how scaling one tier shifts pressure to the next. Record assumptions and revisit them after architecture or traffic changes. A capacity dashboard can create planned work; it should not reuse the urgent page policy without user impact.
Keep dashboard variables bounded as well. A free-form route or customer selector can turn a safe recording rule into an expensive ad hoc query. Offer reviewed service, environment, region, and operation choices, and apply query timeouts.
Review these controls after every major traffic or architecture change. A label and bucket design that was safe for one service can become expensive after new routes, regions, replicas, or tenants are introduced.
When RED shows rising duration or errors and USE shows saturation at a constrained resource, Load Shedding and Adaptive Concurrency Limits turns that evidence into an explicit admission decision before more work reaches the bottleneck. Use Production Overload Troubleshooting when retries, pool wait, or backlog amplify the incident and recovery needs cross-signal verification.
Decision checklist
- Start with user journeys and decisions, not a generic dashboard template.
- Apply RED to request-driven services and USE to constrained resources.
- Use counters, gauges, and histograms according to their semantics.
- Use seconds and other base units consistently.
- Keep route, status, and operation values bounded.
- Reject personal, unbounded, or per-request metric labels.
- Choose histogram buckets around operational objectives.
- Monitor collection freshness and scrape failures.
- Use exemplars only as optional pivots to traces.
- Page on sustained user-impact or rapid SLO burn, not isolated resource noise.
Official sources
- Prometheus instrumentation practices
- Prometheus metric and label naming
- Prometheus histograms and summaries
- Google SRE: Monitoring Distributed Systems
- OpenTelemetry metrics semantic conventions
Official sources accessed August 19, 2026. Confirm metric names, histogram support, and semantic-convention stability against the deployed instrumentation and Prometheus versions.
For Spring services, Spring Boot Actuator, Health, and Metrics Explained applies RED and USE with bounded Micrometer tags, readiness, and liveness boundaries. If those signals show customer harm, Production Spring Boot Incident Troubleshooting prevents a resource symptom from becoming an untested pool, restart, or rollback action.
Related reading
Previous: Structured Logging. Next: Distributed Tracing and Context Propagation. Follow Production Observability & SRE and its topic.
Begin with Observability for Backend Systems, then use SLI, SLO, Error Budgets, and Burn-Rate Alerts to turn these measurements into action. Find the full path in System Design and Topics.
When database duration rises, inspect plan estimates, repeated work, and buffers with PostgreSQL EXPLAIN and Query Plans Explained, while Production Slow Query Troubleshooting connects RED and USE evidence to pool wait, locks, retries, safe mitigation, and a user-centered recovery window.