Quick answer
Structured logging records an event as typed fields rather than an interpolated sentence. A useful production log identifies when and where an event happened, what operation was attempted, what outcome occurred, and which request, trace, span, or message can connect it to other evidence. The schema is an operational interface: field names, types, meanings, redaction, and retention should be reviewed like an API.
Use service.name and deployment.environment.name as resource identity. Use trace_id and span_id to connect a log written inside an active span. Use request_id as an application or edge support identifier when a human needs a compact reference. Use a stable message_id for asynchronous work and retries. These identifiers have different lifetimes; copying one value into every field hides the real boundary.
Do not turn correlation IDs into metric labels. Do not treat an incoming request ID, trace context, or message header as authorization. Never log credentials, session cookies, access tokens, raw payment data, or unbounded request bodies. Prefer a small allowlisted envelope, normalized error classification, and event-specific fields that answer a known operational question.
Learning objectives
- Design a typed logging contract for the Spring order service with bounded event meaning, correlation, redaction, retention, and access.
- Preserve request, trace, message, and retry identity without using correlation identifiers as authorization or metric dimensions.
Prerequisites
Understand Spring Boot, Micrometer, and OpenTelemetry Instrumentation and basic structured JSON logging.
Production failure scenario
The shared incident begins when a slow query in the inventory service exhausts its database connection pool.
The inventory service receives reservation requests from the order API and from an asynchronous consumer. After a deployment, a slow query holds database connections. Pool wait rises, the order API times out and retries, and queue backlog grows. The checkout SLO burns even though many requests eventually succeed.
A useful log sequence preserves boundaries. The order API log has its own request_id and current trace_id. The outgoing inventory call creates a child span_id. When the order service publishes a reservation message, it records the stable message_id. The consumer later receives that message, continues or links trace context according to policy, and logs the same message_id. Every record includes service.name, deployment.environment.name, normalized operation, outcome status, and a route template rather than the raw URL.
The slow path needs facts, not a stack trace on every request. One warning can record operation=inventory.reserve, pool_wait_ms, query_name=reserve_inventory, attempt, and an error classification such as dependency_timeout. A deployment event supplies the version. The operator finds a high-latency trace, pivots to logs by trace_id, sees pool wait dominate, then checks all occurrences of the normalized query name. After rollback, fresh logs show normal pool wait while queue age and SLO burn recover.
Evidence and system boundary
OpenTelemetry’s logs data model distinguishes the log record from its resource and instrumentation scope. Trace and span IDs can correlate a record with tracing data. Semantic conventions provide common attribute names, but stability differs by convention. A log backend may rename or index fields; preserve the canonical meaning at collection so query details do not leak into application code.
W3C Trace Context standardizes traceparent and tracestate, not request_id or message_id. A request ID is an application convention. It can remain stable across retries for a user operation or change per network attempt, but the choice must be explicit. A message ID belongs to the durable message identity and should remain stable across redelivery. Trace identity describes a causal observation, and sampling may mean that a valid trace_id has no stored trace.
Operational logs, audit logs, and security-event logs have different evidence boundaries. Operational logs help diagnose behavior and may be sampled or short-lived. Audit records answer who performed a sensitive business or administrative action and often need stronger integrity and retention. Security-event records support detection and investigation. Linking them is useful, but copying all fields into one unrestricted store defeats least privilege.
Minimal implementation
Start with a JSON envelope:
{
"timestamp": "2026-08-02T14:07:31.284Z",
"severity": "WARN",
"event_name": "inventory_reservation_delayed",
"message": "Inventory reservation exceeded the pool-wait objective",
"service.name": "inventory-service",
"deployment.environment.name": "production",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"span_id": "00f067aa0ba902b7",
"request_id": "req_01J4...",
"message_id": "msg_01J4...",
"route": "/inventory/reservations",
"operation": "inventory.reserve",
"status": "timeout",
"pool_wait_ms": 842,
"query_name": "reserve_inventory",
"attempt": 2,
"error_type": "dependency_timeout"
}Keep message readable, but query fields rather than parsing prose. Use UTC timestamps with an unambiguous format. Store duration in a named unit or a standard semantic field. Record normalized error_type; do not use the entire exception message as a grouping key. Include stack traces only where they change diagnosis, and bound their size.
For synchronous requests, middleware should validate or replace an incoming request_id, put it in scoped context, and return the chosen value to trusted callers. Instrumentation should attach current trace_id and span_id automatically. For asynchronous work, include message_id in message metadata and restore it into consumer logging context. Clear context when a request or delivery finishes so pooled threads and reused async resources do not leak identity into the next operation.
A redaction processor should run before export:
processors:
transform/redact:
error_mode: ignore
log_statements:
- context: log
statements:
- delete_key(attributes, "http.request.header.authorization")
- delete_key(attributes, "session.cookie")
- replace_pattern(body, "(?i)bearer\\s+[a-z0-9._-]+", "Bearer [REDACTED]")Prefer allowlisting over an endless denylist. The example demonstrates pipeline placement, not a complete secret detector. Test the exact Collector distribution and transform syntax used in deployment.
Define severity as an operational decision. ERROR should represent an operation that failed and needs aggregation or action, not every exception that application code catches. WARN can represent degraded but handled behavior such as a retry or pool wait above an objective. INFO records meaningful lifecycle or business transitions. DEBUG is temporary diagnostic detail. If the same failure is logged at client, service, repository, and global handler layers, one incident becomes four errors. Pick the layer that owns the outcome, attach the cause once, and let spans preserve the call graph.
Schema evolution needs compatibility. Adding an optional field is usually safe. Changing duration_ms from number to string or reusing status for a different concept breaks queries. Maintain a small field registry with type, meaning, sensitivity, cardinality expectation, and owner. When replacing a field, emit both old and new names for a bounded migration window, update queries, then remove the old field. Do not leave permanent aliases that double indexing cost.
Exception logging should preserve classification without copying uncontrolled data. Record a stable error type, the operation that failed, whether the error is retryable, and a bounded stack trace when it changes diagnosis. Database constraint names or remote status codes can be useful when allowlisted. Full SQL parameters, HTTP bodies, and third-party response payloads are rarely safe. A fingerprint may help grouping, but it should derive from reviewed structural inputs rather than the raw message.
Asynchronous correlation needs explicit attempt semantics. One durable message_id can have several deliveries. Record delivery attempt, broker destination, consumer group, and normalized disposition such as processed, retry_scheduled, or dead_lettered. If a consumer emits a new event, that event receives a new message ID and records causal linkage through trace context or a separate causation ID. This prevents an investigation from mistaking a downstream event for a redelivery of the upstream one.
Failure modes and trade-offs
Free-form logging makes queries brittle. The strings “timeout calling inventory,” “inventory timeout,” and “deadline exceeded” may represent one error class. A normalized event_name, operation, and error_type preserve grouping while the human message evolves.
Context leakage is subtle. In Java thread pools, stale MDC values can cross requests if cleanup is skipped. In Node.js, a context created outside the expected asynchronous scope may disappear or attach to unrelated work. Framework and OpenTelemetry integrations should own lifecycle where possible; tests must cover concurrent requests and reused workers.
Duplicate IDs can also mislead. Reusing one request_id for every retry may be correct for a user operation but insufficient to distinguish attempts. Add an attempt field or a separate attempt identity. Replacing a stable message_id on redelivery destroys deduplication evidence. Conversely, using message_id as the trace ID conflates durable message identity with an observation that may span multiple retries.
Logging every success at high volume can cost more than the application. Aggregate predictable behavior in metrics. Use traces for cross-service latency. Keep logs for state changes, classified failures, rare decisions, and diagnostic details. Sampling informational logs may be acceptable; sampling audit evidence without an explicit compliance policy is not.
Common misconceptions
- A correlation ID is not proof of identity or authorization.
- JSON syntax does not guarantee a stable or safe log schema.
- Debug level does not permit credentials, personal data, or raw payload collection.
- Missing logs can reflect pipeline loss rather than missing application activity.
Security, privacy, and cost
The safest sensitive field is one never collected. Do not log passwords, access or refresh tokens, API keys, cookies, authorization headers, private keys, full payment details, or raw request bodies. Avoid personal data in trace_id, tracestate, baggage, and custom correlation IDs. Hashing an identifier does not automatically make it anonymous; stable hashes can still enable tracking and may be reversible through a small input space.
Classify personally identifiable information (PII) before defining the envelope. Names, email addresses, postal addresses, account identifiers, device identifiers, and precise locations may identify a person directly or in combination. Prefer an operational classification such as customer_lookup_failed over copying the value that failed. When a narrowly justified field must be retained, document purpose, access, retention, deletion, and regional constraints rather than relying on redaction after broad collection.
Treat log fields received from clients as untrusted. Structured encoders prevent newline injection from forging separate records, but dashboards and export formats still need escaping. Limit field length and collection depth. Apply tenant-aware authorization at query time and audit access to sensitive indexes.
Index only fields needed for operations. service.name, environment, severity, event_name, operation, error type, and bounded correlation fields are common candidates. High-cardinality indexes increase cost. Retain debug records for less time than security or audit evidence. Document deletion and legal-hold behavior rather than assuming one global retention period.
Testing and validation
Use golden fixtures for the envelope. Assert types, required fields, UTC timestamp parsing, maximum lengths, and the absence of forbidden keys. Run concurrent requests and confirm each log keeps the correct request_id, trace_id, and span_id. Redeliver one message and prove the stable message_id remains while attempt data changes.
Send secret-shaped test data through the whole pipeline and inspect exported output. Include mixed-case authorization headers, tokens in exception messages, multiline user input, nested payloads, and long values. A redaction unit test is not enough if a later processor copies the original body into another field.
Create a sampled trace and verify that correlated logs remain understandable when the trace is not retained. Simulate log-backend rejection and monitor Collector retries, queue utilization, refused records, and drops. Validate that absence of logs cannot be confused with absence of failures by keeping user-impact metrics independent.
During the shared incident drill, start with the SLO alert, locate a slow trace, pivot to the structured warning by trace_id, group by query_name, and confirm the deployment version. After mitigation, check that new warnings stop and that pool wait, queue backlog, and the SLO recover.
Test schema migration with old and new producers active simultaneously. Queries and alerts should tolerate the bounded overlap without double-counting. Verify parsers reject or quarantine records with the wrong type instead of silently converting pool_wait_ms="slow" to zero. Track malformed-record counts and sample safe metadata for debugging.
Measure query usefulness as well as ingestion. On-call exercises should locate one request, all attempts for one message_id, and all failures for one normalized operation within the expected response time. If a lookup requires scanning unindexed raw text, redesign the schema or index policy before an incident forces an expensive emergency change.
Document support workflows for correlation IDs. A customer-facing request_id should lead to a bounded internal search without exposing trace internals to the caller. Define its retention window, format validation, and response policy. If the referenced logs have expired, support must report that boundary instead of searching broader sensitive datasets. Test the workflow with expired IDs, malformed input, and concurrent requests so convenience never becomes an authorization bypass.
Include schema ownership in service onboarding. New services should emit a canary record, prove redaction, and demonstrate trace-to-log lookup before production traffic. This catches missing resource identity and incompatible field types while the blast radius is small.
Decision checklist
- Publish and version a small log schema.
- Separate resource identity from event-specific attributes.
- Define the lifecycle of request, trace, span, and message identity.
- Use structured encoders and scoped context cleanup.
- Normalize event names, operations, outcomes, and error classes.
- Allowlist fields and redact before export.
- Keep metric labels separate from log correlation fields.
- Separate operational, audit, and security retention policies.
- Monitor log-pipeline freshness, rejection, queueing, and drops.
- Test queries and runbooks against real exported fixtures.
Official sources
- OpenTelemetry logs data model
- OpenTelemetry logs semantic conventions
- OpenTelemetry baggage security considerations
- W3C Trace Context
- OWASP Logging Cheat Sheet
Official sources accessed August 19, 2026. Verify the stability level of individual OpenTelemetry log attributes against Semantic Conventions 1.44.0 before treating them as durable contracts.
Spring operators can pair these protected correlation facts with the low-cardinality metrics and minimal management surface in Spring Boot Actuator, Health, and Metrics Explained. Production Spring Boot Incident Troubleshooting then keeps a single log event from being mistaken for population evidence or permission to restart, retry, or alter durable orders.
Related reading
Previous: Spring Boot Instrumentation. Next: RED, USE, and Golden Signals. Follow Production Observability & SRE and its topic cluster.
Use Observability for Backend Systems to decide which facts belong in logs, then connect them across services with Distributed Tracing and Context Propagation. Continue through System Design or browse the Topics cluster.