Quick answer
Telemetry is operational evidence only when producers and consumers agree on identity, meaning, units, time, and expected delivery. A graph with no points can mean zero traffic, a renamed attribute, a failed exporter, an expired series, or a query bug. Treat every signal schema as a versioned interface and monitor its chain of custody.
For the Spring order service, stable resource identity connects API, inventory, database, Kafka, and Kubernetes evidence. Low-cardinality attributes such as route template, outcome class, environment, and version support aggregation. Order IDs, customer IDs, raw URLs, exception messages, and trace IDs do not belong in metric dimensions. They may appear only in protected, bounded logs or traces under explicit privacy and retention policy.
Learning objectives
- Define a versioned telemetry contract for resource identity, signal meaning, units, timestamps, and bounded attributes.
- Detect missing, delayed, duplicated, renamed, or high-cardinality evidence before using telemetry to make an incident decision.
Prerequisites
Be comfortable with HTTP services, Spring Boot, basic metrics, logs, traces, and the evidence model from Observability for Backend Systems.
Evidence and system boundary
An application observation passes through instrumentation, an SDK or registry, processors, a Collector, an exporter, a backend, and a query. Each boundary can filter, aggregate, sample, delay, duplicate, rename, or reject data. The backend view is therefore not the application itself.
OpenTelemetry Specification 1.60.0 and Semantic Conventions 1.44.0 distinguish Stable and Development contracts. A stable API does not make every instrumentation package or semantic-convention group stable. Record the exact convention version used by producers and dashboards, and require migration tests before enabling a new schema.
telemetry_contract:
resource:
service.name: order-service
deployment.environment.name: production
service.version: "example-build-digest"
bounded_dimensions: [http.route, http.request.method, outcome]
protected_correlation: [trace_id, request_id, message_id]
forbidden_metric_dimensions: [order_id, customer_id, raw_url, exception_message]The values are explanatory examples. They contain no real environment, user, credential, or order data.
Schema, time, and quality controls
Define names, types, base units, monotonicity, temporality, histogram boundaries, and allowed attributes. Reject mixed seconds and milliseconds before a dashboard combines them. Preserve event time and observed time separately when queues or retries can delay delivery. Document whether duplicate log or span delivery is possible and make consumers tolerant where appropriate.
Track freshness and expected presence with independent health signals. For a request-driven service, compare known synthetic traffic with scrape or export evidence. For the Collector, inspect accepted, refused, queued, retried, failed, and dropped items. A telemetry quality objective protects diagnosis; it is not the product SLO.
Cardinality is the number of distinct label combinations. One order_id label creates a potential series per order. Raw paths such as /orders/84721 have the same problem. Use /orders/{id} for aggregate metrics and pivot to a protected trace or structured log only after the aggregate signal bounds the incident.
Production failure scenario
A Spring order service rollout updates HTTP semantic conventions. The dashboard query still groups by the retired attribute name, so checkout traffic appears to fall to zero. At the same time the Collector exporter queue grows. Operators initially interpret the empty graph as recovery.
The safe investigation compares synthetic checkout results, application emission, Collector receive/export counters, backend freshness, and the instrumentation version. The team repairs the query and schema rollout, then verifies fresh user outcomes. It does not infer that missing telemetry meant missing orders.
Common misconceptions
- A stable OpenTelemetry API does not make every semantic convention or contrib instrumentation stable.
- More fields do not automatically produce more useful evidence; unbounded fields can destroy the aggregate system.
- A trace ID is a correlation key, not an authorization credential or safe metric label.
- A zero, an absent series, and a failed query have different meanings.
- Schema compatibility must include dashboards, alerts, recording rules, and downstream exports—not only application compilation.
Decision checklist
- Name an owner for every telemetry schema and consuming alert.
- Pin or record instrumentation and semantic-convention versions.
- Keep resource identity consistent across signals.
- Use base units and explicit types.
- Bound metric and indexed-log dimensions.
- Test redaction before export.
- Monitor freshness, rejects, retries, drops, and query errors.
- Compare telemetry with an independent user-impact signal.
- Roll schema changes through producers and consumers with compatibility evidence.
Related reading
Previous: Observability for Backend Systems. Next: Spring Boot, Micrometer, and OpenTelemetry Instrumentation. Continue through Production Observability & SRE and its topic cluster.
Connect the same identity rules to Kubernetes Observability and Debugging and Structured Logging and Correlation IDs.
Official sources
- OpenTelemetry Specification 1.60.0
- OpenTelemetry Semantic Conventions 1.44.0
- OpenTelemetry versioning and stability
- Prometheus instrumentation practices
Official sources accessed August 19, 2026. Recheck stability tables before adopting a Development convention as a durable interface.