Distributed Tracing and Context Propagation Explained

Quick answer

Distributed tracing follows one operation across process, service, database, and messaging boundaries. A trace contains spans representing timed operations and causal relationships. Context propagation carries the current trace identity between components. W3C Trace Context standardizes the HTTP traceparent and tracestate headers so different tracing systems can interoperate.

Tracing is useful when aggregate metrics show that checkout is slow but cannot identify which dependency owns the delay. A trace can show the order API waiting on inventory, the inventory service waiting for a database connection, and an asynchronous reservation span delayed behind a queue backlog. Structured logs correlated with trace_id and span_id provide detailed facts. A stable message_id remains the durable identity across redelivery even when trace relationships or attempts differ.

Propagation is not trust. Validate incoming context, define where to restart or sanitize it, and never put personal data in tracestate or blindly trust baggage. Sampling also creates an evidence boundary: an unsampled or dropped trace does not prove the request never happened. Metrics and SLOs remain the complete aggregate control signal.

Learning objectives

• Propagate W3C trace context through the Spring order service’s HTTP, asynchronous, database, and messaging boundaries.
• Choose span parents, links, baggage, and sampling while preserving trust, privacy, and incomplete-evidence boundaries.

Prerequisites

Understand RED, USE, and Golden Signals, request/message identity, and basic distributed request flows.

Production failure scenario

The shared incident begins when a slow query in the inventory service exhausts its database connection pool.

An inventory query becomes slow after deployment and holds connections longer. The inventory service connection pool saturates. The order API waits, times out, and retries. Reservation messages accumulate, producing a queue backlog. Checkout latency rapidly consumes the SLO while the top-level order handler appears mostly healthy.

A representative trace begins at the order API. The server span carries service.name=order-service and deployment.environment.name=production. It creates a client span for the inventory HTTP call. The inventory server span creates a database span and records pool-wait evidence separately from query execution. The trace shows that most duration occurs before the query obtains a connection, preventing a false conclusion that network latency is responsible.

For asynchronous work, the producer injects context into message metadata and records stable message_id. The consumer extracts that context and creates a consumer span. Whether the consumer span is a child or a link depends on the messaging semantics, fan-out, batching, and delay. Redelivery keeps the same message_id but can create a new processing span so each attempt remains observable. Logs during processing include trace_id, span_id, request_id when relevant, and message_id.

Evidence and system boundary

A version 00 traceparent has four fields:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

They represent version, trace ID, parent ID, and trace flags. The sampled flag communicates a recording decision preference; it does not guarantee that every span reaches storage. tracestate carries vendor-specific trace information and must be handled with its ordering and size rules. Invalid traceparent must not be partially trusted, and invalid context may cause a new trace to start.

OpenTelemetry propagators serialize and deserialize span context and baggage. Baggage is a separate key-value store that can cross service boundaries. It is not automatically attached to telemetry and has no built-in integrity check. External services may receive automatically propagated baggage, so define an allowlist and remove sensitive or untrusted entries at trust boundaries.

Span semantics matter. A server span represents receiving a request; a client span represents an outgoing request; producer and consumer spans describe messaging operations; an internal span covers meaningful in-process work. Avoid creating a span for every helper function. Name spans with stable operations, not raw URLs, SQL values, user identifiers, or message IDs.

Minimal implementation

For HTTP, accept valid W3C context, create a server span, and inject the current context into each outgoing request. Keep application support identity separate:

HTTP request
  traceparent: 00-<trace-id>-<caller-span-id>-01
  tracestate: vendor=value
  x-request-id: req_01J4...

The receiving service should expose request_id in logs or trusted responses according to policy, but tracing libraries should own trace_id and span_id. Do not generate a trace ID by hashing a user or order identifier.

For messaging, use explicit metadata:

{
  "message_id": "msg_01J4R7V2KQ2N",
  "operation": "inventory.reserve",
  "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-7a085853722dc6d2-01",
  "tracestate": "vendor=value"
}

Keep message_id stable across broker redelivery. Create a new consumer span per attempt and record retry attempt as a bounded attribute. For batch consumption, links can represent multiple producer contexts without pretending the batch has one parent. For long queue delays, record or derive publish-to-consume latency without keeping a producer span open for hours.

Add a few reviewed span attributes: route template, normalized operation, bounded status, database system, and peer service identity where stable conventions permit. Record exceptions according to OpenTelemetry conventions, set status based on the operation outcome rather than every caught exception, and avoid duplicate error events from multiple instrumentation layers.

Head sampling decides near trace creation. It is cheap and predictable but cannot know the final latency or error. Tail sampling evaluates completed or partially completed traces in a Collector and can retain errors or slow traces, but it requires consistent routing, memory, wait time, and policies for incomplete traces. Preserve aggregate metrics independently because neither strategy produces a complete dataset.

Model database wait and execution separately when instrumentation permits it. A single database client span that includes pool acquisition and query time tells the operator the dependency is slow but not why. An internal pool-acquisition span or event, plus a database span with a stable operation name, distinguishes saturation from an inefficient query. Do not attach SQL parameters. A normalized statement name or reviewed query fingerprint can support comparison without leaking values.

Messaging traces need time semantics. Producer send duration is not queue delay. Consumer processing duration is not end-to-end freshness. Record enqueue time through trusted broker metadata or a bounded application field, then calculate queue age at consumption. Clock skew can make naive subtraction negative, so prefer broker-provided timestamps or synchronized systems and handle invalid values. Long delays also mean the original producer trace may be unavailable by the time the consumer runs.

Trace status should describe the operation outcome, not every handled exception. A retryable attempt can record an exception and still lead to a successful overall request. Marking every child span and parent as error multiplies counts; marking none hides degradation. Define which layer owns the final outcome and preserve attempt information as events or attributes. Metrics remain the authority for aggregate error rate.

Propagation across scheduled work also needs design. A background job created from a request may run minutes later. Continuing one trace can create extremely long traces and retention mismatches. Starting a new trace with a span link to the originating context often gives clearer lifecycle evidence. Preserve durable job or message_id identity separately so correlation survives after the originating trace expires.

Failure modes and trade-offs

Broken propagation creates separate traces for one request. Common causes include manual HTTP clients, background executors, unsupported message metadata, context cleanup bugs, proxies that remove headers, or code that starts work after the parent scope closes. A trace can also look broken when sampling or backend ingestion drops spans. Test propagation and pipeline health before blaming application causality.

Wrong parentage creates a convincing but false graph. A redelivered message should not appear as one endlessly running consumer span. Batch operations should not arbitrarily choose the first message as the parent of all others. Use span links when work relates to several independent contexts or when an attempt relationship is non-hierarchical.

Over-instrumentation increases cost and noise. Spans for getters, JSON parsing, and trivial wrappers hide remote calls and queue waits. Under-instrumentation leaves one giant server span. Instrument boundaries where ownership, latency, or failure behavior changes.

Tail sampling is not free reliability. A central sampler can become a bottleneck. If traces for one ID reach different collectors, the decision may be incomplete. Persistent queues can reduce transient loss but do not make sampling or export infallible. Monitor dropped, refused, late, and decision-timeout counts.

Common misconceptions

• A trace context header is not trusted identity.
• Baggage is not a safe place for credentials or personal data.
• A parent-child tree is not always correct for queues, batches, fan-in, or redelivery.
• Retained sampled traces are not a complete or necessarily unbiased request population.

Security, privacy, and cost

Treat incoming traceparent, tracestate, baggage, and custom request IDs as untrusted. Validate format and length. Decide whether to continue, sanitize, or restart traces at public ingress and third-party egress. W3C guidance forbids personally identifiable information in tracestate. Do not put credentials, authorization decisions, user IDs, order IDs, or raw query values into propagating fields.

Baggage can silently cross more services than intended. Automatic instrumentation may forward it to third parties. Use a narrow allowlist, strip it at boundaries, and never use an unsigned baggage value as proof of tenant, role, or account identity.

Control span volume with meaningful boundaries and sampling. High-cardinality attributes increase index cost even when trace count is bounded. Keep sensitive attributes out of the pipeline rather than relying only on backend access controls. Define retention based on operational need and incident response, not indefinite convenience.

Testing and validation

Build a propagation contract with an order API, fake inventory service, and fake queue consumer. Assert one valid HTTP trace relationship, one message link or parent relationship according to the chosen convention, and consistent service.name and deployment.environment.name. Confirm logs during each span contain matching trace_id and span_id, while message_id remains stable across redelivery.

Test malformed, oversized, and unsupported-version traceparent values. Test invalid tracestate, external baggage, missing context, and a proxy that strips headers. Verify the service follows documented trust policy and never authorizes from telemetry context.

Exercise head sampling at zero, partial, and full rates. Exercise tail policies for errors, slow traces, and normal traffic. Confirm SLO metrics remain correct regardless of tracing decisions. Disable the trace backend and observe Collector queue, retry, refusal, and drop signals.

During the shared incident, select an exemplar or trace from the high-latency bucket, find pool wait and database duration, pivot to structured logs, and compare deployment version. After rollback, inspect fresh traces rather than assuming old trace completion proves recovery.

Verify trace topology, not merely the presence of IDs. A test should assert the order server span owns the inventory client span, the inventory server span follows the propagated parent, and database work sits beneath inventory. Messaging tests should assert the chosen parent or link semantics and one new consumer span per attempt. This catches instrumentation that copies a trace ID but produces unrelated roots.

Run mixed-version tests during instrumentation upgrades. Old and new services may emit different semantic attributes or propagation formats. Confirm the supported propagator set, precedence, and migration window. Avoid enabling several propagators without understanding header conflicts, because duplicate context can create separate or incorrectly joined traces.

Measure trace completeness for synthetic canaries. The canary knows the expected services and boundaries, so a missing span becomes a pipeline signal. Do not generalize that percentage to all production traffic without accounting for sampling, optional branches, and backend retention.

Trace retention must match operational questions. Short retention may be sufficient for active incidents but insufficient for delayed message investigation. Keeping every trace indefinitely is costly and increases privacy exposure. Choose retention and sampling together, preserve aggregate metrics for long windows, and retain selected incident evidence through an audited process. Queries and runbooks should state the oldest trace they expect to find so missing historical data is not misdiagnosed as propagation failure.

When a trace crosses organizations, minimize shared metadata and document responsibility for header handling. Third parties may ignore, replace, or return context. Treat that boundary as an external dependency and verify correlation with contract tests rather than assumptions.

Record the tested propagation matrix in service documentation, including HTTP clients, proxies, brokers, scheduled workers, and external boundaries. This makes an upgrade review concrete and prevents unsupported paths from being mistaken for random trace loss.

Decision checklist

• Standardize on W3C Trace Context for interoperable HTTP propagation.
• Keep request, trace, span, and message identities semantically separate.
• Use stable operation-based span names and bounded attributes.
• Choose parent versus link semantics for messaging and batches explicitly.
• Sanitize context and baggage at trust boundaries.
• Never authorize from trace or baggage values.
• Document head and tail sampling policies and bias.
• Monitor missing spans, late traces, sampler pressure, export failure, and drops.
• Keep SLO metrics independent from trace sampling.
• Test propagation after framework, proxy, broker, or instrumentation upgrades.

Official sources

• W3C Trace Context
• OpenTelemetry context propagation
• OpenTelemetry traces
• OpenTelemetry baggage
• OpenTelemetry sampling

Official sources accessed August 19, 2026. Check current semantic-convention stability and messaging span guidance for OpenTelemetry 1.60.0 and Semantic Conventions 1.44.0.

Related reading

Previous: RED, USE, and Golden Signals. Next: JVM Profiling and JFR. Follow Production Observability & SRE and its topic.

Use Structured Logging and Correlation IDs for trace-to-log pivots and OpenTelemetry Collector Pipeline for transport, processing, and tail-sampling boundaries. Follow the ordered path in System Design or Topics.