OpenTelemetry Collector Pipeline Explained

Quick answer

The OpenTelemetry Collector is a vendor-neutral service that receives telemetry, processes it, and exports it to one or more backends. A pipeline combines receivers, processors, exporters, and optional connectors for traces, metrics, or logs. The Collector separates application instrumentation from storage vendors and gives operators one place to enforce batching, memory protection, redaction, routing, sampling, retry, and export policy.

A production pipeline is bounded and observable. memory_limiter protects the process before batching. batch improves export efficiency. Sensitive attributes are removed before export. Exporters use retry_on_failure and a bounded sending_queue for transient failures. These mechanisms improve resilience but do not guarantee no telemetry loss. Invalid data, exhausted queues, full disks, crashes, permanent backend rejection, and configuration mistakes can still lose evidence.

Applications send stable resource identity such as service.name and deployment.environment.name. Logs and traces may include trace_id, span_id, request_id, and message_id; metrics use low-cardinality route, status, and operation dimensions instead of per-request identifiers. Monitor the Collector itself as a production component.

Learning objectives

• Design a bounded OpenTelemetry Collector pipeline for the Spring order service with explicit receive, process, queue, retry, redact, and export behavior.
• Use Collector self-observability to distinguish application health from delayed, rejected, dropped, or unavailable telemetry.

Prerequisites

Complete JVM Profiling and JFR and understand OTLP plus the application’s signal ownership.

Production failure scenario

The shared incident begins when a slow query in the inventory service exhausts its database connection pool.

An inventory query becomes slow and holds database connections. Pool wait rises. The order API retries, increasing traffic, while reservation messages form a queue backlog. Checkout latency consumes the SLO. At the same moment, trace volume grows because slow and failed operations generate more telemetry.

If the Collector is unbounded, the incident can take down the evidence pipeline. Export latency rises, batches accumulate, memory grows, and the process may be killed. If the queue is too small, traces and logs drop before operators see the cause. If it is too large, the Collector may delay data until the alert and dashboard look stale. Pipeline health must be visible alongside application health.

The evidence contract passes through the pipeline without changing meaning. service.name and deployment.environment.name identify resources. trace_id and span_id connect traces and logs. request_id and message_id support application and asynchronous correlation. Processors may remove unsafe fields or normalize bounded attributes, but they must not convert high-cardinality identity into metric labels.

Evidence and system boundary

The Collector supports multiple receivers and exporters, but OTLP is the OpenTelemetry protocol designed for telemetry transfer. A receiver accepts data. A processor transforms, filters, batches, samples, or protects resources. An exporter sends data to a backend. Pipelines are signal-specific: a component must be listed in the service pipeline before it becomes active.

The Collector is not a telemetry database. An exporter success usually means the configured destination accepted the export according to its protocol, not that every future query will return the data forever. Retry policies normally apply only to retryable failures. Permanent validation errors should not loop indefinitely. A sending queue buffers work in memory or configured persistent storage according to the exporter helper and distribution.

Collector components evolve independently and can have different stability levels. Validate configuration with the exact Collector distribution and version being deployed. Do not copy a component name from a contrib example and assume it exists in a core or vendor distribution.

Minimal implementation

This example receives OTLP over gRPC and HTTP, applies memory protection, removes sensitive attributes, batches data, and exports through OTLP. Environment variables supply endpoints and credentials rather than committed secrets:

extensions:
  health_check: {}

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
      http:
        endpoint: 127.0.0.1:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  transform/redact:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - delete_key(attributes, "http.request.header.authorization")
          - delete_key(attributes, "session.cookie")
    log_statements:
      - context: log
        statements:
          - delete_key(attributes, "http.request.header.authorization")
          - delete_key(attributes, "session.cookie")
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlp/backend:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      authorization: ${env:OTEL_EXPORTER_OTLP_AUTHORIZATION}
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      queue_size: 5000
      num_consumers: 8

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, transform/redact, batch]
      exporters: [otlp/backend]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/backend]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, transform/redact, batch]
      exporters: [otlp/backend]

The order matters. Memory limiting should occur early. Redaction must happen before data leaves the trust boundary. Batching happens after transformations. The loopback receiver endpoints are safe only when applications share the host or network namespace; a remote agent or gateway needs an explicitly protected listener instead. The exact transform paths and exporter authentication syntax must be tested against the selected distribution.

Choose a deployment pattern deliberately. An agent Collector near each workload reduces direct application-to-backend coupling and can enrich local resource data. A gateway tier centralizes sampling, routing, and credentials. A combined pattern uses agents for collection and gateways for organization policy. Tail sampling may require trace-aware load balancing so spans from the same trace_id reach the same decision point.

Expose Collector internal telemetry to a separately dependable monitoring path. Track accepted, refused, sent, failed, retried, queued, and dropped telemetry; process memory and CPU; queue capacity; export latency; configuration reload or restart events; and receiver availability. A health endpoint proves the process responds, not that all pipelines deliver fresh data.

Plan capacity from peak incident volume, not average daytime traffic. Estimate spans, metric points, and log records per second; average encoded size; batch size; export latency; and outage buffer. Leave headroom for retries and tail-sampling state. Load-test with the same processors because transforms, regex redaction, and sampling policies can dominate CPU. A gateway that is comfortable at normal volume may fail exactly when an incident increases error logs and spans.

High availability requires clear delivery semantics. Multiple stateless gateways behind a load balancer can receive OTLP, but tail sampling needs trace affinity or a trace-aware load-balancing exporter. Persistent queues survive some process restarts but do not automatically replicate across nodes. If a node and its disk fail, buffered telemetry may be lost. Document whether the platform optimizes for availability, cost, sampling correctness, or durability rather than promising all four.

Configuration promotion should be transactional from the operator’s perspective. Validate syntax and component availability, start a canary, send synthetic telemetry, compare accepted and exported counts, then expand. Keep the previous version and configuration ready for rollback. A malformed filter can drop one signal while health checks remain green. Synthetic canaries should exercise traces, metrics, logs, redaction, and expected routing.

Backpressure needs an end-to-end contract. A receiver refusal may cause an SDK exporter to retry or drop according to its own queue and timeout. Cascading queues can hide an outage for hours and then deliver stale telemetry in a burst. Set compatible limits, monitor age as well as depth, and define which layer is allowed to shed load. Protect application latency: telemetry export must not block critical business work indefinitely.

Failure modes and trade-offs

Unbounded memory is the most visible failure. Batches and exporter queues grow during backend slowdown. memory_limiter introduces backpressure or refusal before the process exceeds its budget. Applications and upstream Collectors must handle that signal; otherwise refusal becomes silent loss.

Retry storms can extend an incident. Many Collectors retrying at the same interval overload a recovering backend. Use exponential backoff and bounded elapsed time where supported. A queue absorbs short outages but adds latency. Persistent queues improve crash survival but require disk capacity, permissions, encryption decisions, and monitoring.

Processor ordering can leak data. Redacting after an exporter is impossible. A routing connector or debug exporter can create an unintended copy. Review every pipeline and every exporter, not only the primary traces path. Avoid debug verbosity in production because it can expose telemetry payloads.

Configuration drift can make one environment incomparable. If staging drops attributes that production retains, tests may miss privacy leaks or query failures. Version configuration, validate it in CI, and promote the same policy with environment-specific endpoints and credentials.

Tail sampling trades completeness, latency, memory, and cost. Policies that retain errors and slow traces are useful for diagnosis but bias analysis. Do not compute a traffic SLI from a tail-sampled trace set. Keep aggregate request metrics before sampling.

Common misconceptions

• A retry queue does not guarantee lossless telemetry delivery.
• A Collector is not the observability backend or business source of truth.
• Batching cannot create missing semantic identity.
• More queue capacity can delay failure and consume memory without fixing an unavailable exporter.

Security, privacy, and cost

Bind OTLP receivers only to intended networks and use transport authentication where the environment requires it. A public unauthenticated receiver is an ingestion and cost-abuse risk. Store exporter credentials in a secret manager or environment injection, not source control. Rotate them and restrict them to the necessary destination.

Apply data minimization before export. Delete authorization headers, cookies, tokens, raw request bodies, SQL values, and personal data. Baggage and resource attributes need the same review as log bodies. Redaction patterns are defense in depth; allowlisted instrumentation is stronger.

Queues consume memory or disk, and every copied exporter multiplies volume. Set telemetry budgets by service and signal. Keep metric cardinality bounded. Sample traces with documented bias. Reduce repetitive logs. Preserve the SLO and incident signals needed to prove user impact and recovery.

Testing and validation

Validate configuration with the actual Collector binary before deployment. Send one trace, metric, and log through OTLP and verify all pipelines activate. Confirm required fields survive: service.name, deployment.environment.name, trace_id, span_id, request_id, and message_id where relevant. Confirm forbidden sensitive fields do not reach a capture exporter.

Test backend outage behavior. Observe retry timing, sending_queue growth, memory_limiter actions, refused telemetry, and eventual drops. Fill the queue deliberately in a safe environment. Restart the Collector and verify the expected behavior for in-memory or persistent queues. Test permanent invalid-data rejection separately from retryable unavailability.

Measure freshness from application emission to backend query. A green health check with stale data is a failed pipeline. Alert on sustained export failure, high queue utilization, refused records, drops, restart loops, and missing expected telemetry.

During the shared incident drill, increase trace and log volume while the backend is slow. Verify the Collector stays within memory limits and preserves the chosen high-value evidence. After rollback, confirm queue age drains and current telemetry appears promptly rather than reading delayed incident data as a new failure.

Test configuration reload and restart behavior. Determine whether the deployment method replaces processes, drains queues, or interrupts receivers. Send canary telemetry immediately before and after the change, then check duplicates, gaps, and export delay. A health endpoint that stays available behind a load balancer can conceal one failing replica.

Validate tenant and environment routing with synthetic attributes that contain no real customer data. Production telemetry must not reach a development backend, and one tenant’s credentials must not authorize another route. Test the default path for missing or malformed routing attributes; silently sending unknown data to the broadest destination is unsafe.

Measure processor cost individually when possible. Regex-heavy redaction, attribute transforms, and tail sampling can consume far more CPU than batching. Benchmark representative payload sizes and hostile long values. Bound transform input and set error_mode deliberately so malformed data cannot crash the entire pipeline or bypass required redaction.

Keep a loss ledger for drills and incidents: application exports attempted, receiver accepted, processor refused, exporter sent, backend accepted when observable, and query-visible records. The counts may not align perfectly across protocols, but unexplained gaps identify the boundary that needs better instrumentation.

Test version upgrades with representative captured shapes that contain synthetic, non-sensitive values. Validate the configuration against the target binary, compare component stability levels, and inspect release notes for renamed settings or changed defaults. A configuration that parses can still change retry timing, queue persistence, attribute transformation, or metric names. Promote one canary Collector first and compare acceptance, drop, latency, and resource metrics before completing the rollout.

Document ownership across application, platform, and backend teams. Application owners control instrumentation semantics; platform owners control Collector availability and policy; backend owners control ingestion and query service. One end-to-end SLO for telemetry freshness needs a named owner for incident coordination even when several teams contribute. Test escalation paths during drills, because a pipeline failure that crosses ownership boundaries can otherwise remain unresolved while each component appears locally healthy.

Back up configuration through version control, not exported telemetry payloads. Recovery should recreate receivers, processors, exporters, secrets references, and routing from reviewed artifacts. Test that rollback restores delivery without reintroducing a redaction or cardinality defect.

Keep a tested emergency mode that reduces optional telemetry volume without disabling SLO metrics or required security evidence. Activating it should be auditable, time-bounded, and reversible after backend capacity recovers.

Decision checklist

• Pin and inventory the Collector distribution and component versions.
• Validate configuration with the deployed binary.
• Put memory_limiter early and batch after transformations.
• Redact before every exporter and review alternate routes.
• Bound retries and sending queues; document expected loss behavior.
• Secure receivers and inject exporter credentials from secrets.
• Monitor accepted, refused, queued, failed, retried, dropped, and stale data.
• Keep SLO metrics independent from trace sampling.
• Capacity-test incident volume and backend outages.
• Maintain a rollback for Collector configuration and version changes.

Official sources

• OpenTelemetry Collector documentation
• OpenTelemetry Collector configuration
• OpenTelemetry Collector resiliency
• OpenTelemetry Collector internal telemetry
• OpenTelemetry security guidance

Official sources accessed August 19, 2026. Component availability and configuration fields must be checked against the exact Collector distribution and OpenTelemetry 1.60.0 contracts used for implementation.

Related reading

Previous: JVM Profiling and JFR. Next: SLIs, SLOs, Error Budgets, and Burn-Rate Alerts. Follow Production Observability & SRE and its topic.

Connect the pipeline to Distributed Tracing and Context Propagation, and monitor its user value with SLI, SLO, Error Budgets, and Burn-Rate Alerts. Continue via System Design or Topics.