Dead Letter Queue Explained | Backend Study Lab

Quick answer

A dead-letter queue quarantines messages that normal processing has stopped attempting. A reliable design classifies failures, bounds transient retries, preserves safe metadata, alerts an owner, and requires an inspected decision to replay or discard.

Keep the original stable message ID through retry and replay. If a broker assigns a new transport ID, carry the original as metadata or causation_id. Every replay must pass through the same atomic, idempotent consumer, preventing duplicate inventory reservations or charges.

Broker features differ. RabbitMQ-specific: a queue can republish to a configured dead-letter exchange. Amazon SQS-specific: a redrive policy moves repeatedly received messages to a DLQ after maxReceiveCount. Applications still own classification, remediation, approval, redaction, and idempotency.

Failure timeline

Use one order flow: OrderPlaced reserves inventory, InventoryReserved authorizes payment, PaymentAuthorized sends a notification.

Inventory receives msg-847, attempts a PostgreSQL reservation, and the database connection times out.
The consumer records attempt one and schedules a delayed retry with the same msg-847.
Attempts two and three fail with the same transient class, with increasing delay and jitter.
The retry budget is exhausted. The consumer writes or routes a redacted recovery envelope and settles the source delivery only after durable recovery acceptance.
An alert identifies the owning inventory team, queue age, failure growth, and affected order count.
An operator inspects the message and trace, discovers a deployed schema mismatch, and links the remediation change.
A second authorized operator approves a small replay batch. Replay retains msg-847.
The repaired inventory consumer executes its normal idempotent transaction. If the original attempt actually committed before its timeout, the processed_message unique constraint turns replay into a duplicate no-op.
After observed success, the operator expands replay velocity. Payment and notification continue with their own stable IDs.

A timeout cannot prove PostgreSQL rolled back, and a lost confirmation cannot prove broker rejection. Recovery preserves identity and assumes duplicates.

Classify failures before retrying

Classification determines whether time can plausibly change the result:

Transient failure: timeout, connection reset, 429, or temporary dependency unavailability. Retry within a bounded budget.
Permanent failure: unsupported business state, revoked account, or nonexistent product. Quarantine it; repetition adds load.
Poison message: validly encoded input that deterministically crashes or violates a handler assumption. Isolate it so one record cannot block healthy work.
Malformed message: invalid JSON, missing ID, wrong type, or schema violation. Do not deserialize repeatedly; store redacted input and errors.
Expired failure: the business deadline passed. A promotion reservation or same-day notification can become harmful even if processing would now succeed.
Policy failure: security, compliance, tenant, region, or replay-policy rules forbid execution. Route to a restricted workflow rather than retrying around the policy.

Classification must be testable. Unknown failures get a small retry budget, then review. Payment ambiguity needs a provider lookup using the original idempotency key, not another charge.

Bounded retries and dead-letter routing

A retry policy needs maximum attempts and elapsed time, backoff, jitter, per-attempt timeout, and a terminal action. Inventory may try three times over ten minutes; a notification may try six times over a day. One layer owns retries because nested policies multiply attempts.

Attempt count is durable state, carried in trusted attributes or PostgreSQL. Reject retries after the business deadline. A circuit breaker can pause calls without consuming each message’s budget.

Settlement must be unambiguous in code:

Success: commit business state, then acknowledge or delete the delivery.
Retry scheduled: obtain durable, confirmed acceptance of the retry copy, then settle the current delivery.
Terminal failure: obtain durable acceptance of the recovery envelope, then settle the current delivery.
Recovery or retry publication uncertain: do not acknowledge; allow redelivery with the same identity.

Ambiguous confirmation can create duplicate recovery copies. That is safer than silent loss, so the recovery store and consumer must deduplicate.

Preserve recovery metadata safely

Store investigative context without making the DLQ a secret archive. Include:

original message ID and event type;
source destination and original enqueue time;
failure class and a stable error code;
attempt count and retry policy version;
first and last failure times;
trace context, such as traceparent, and a correlation ID;
payload schema version, tenant-safe business reference, and payload hash;
redacted payload or a restricted encrypted reference;
owner, status, remediation link, approval actor, and replay audit fields.

Never copy authorization headers, cookies, API keys, card data, passwords, raw access tokens, or full exception dumps containing SQL parameters. Apply an allowlist serializer before the recovery boundary, replace sensitive fields with "[REDACTED]", limit strings and stack traces, encrypt stored data, and restrict read and replay permissions. Hashing a low-cardinality secret is not redaction.

PostgreSQL can provide the durable state marker for the operated workflow:

CREATE TABLE message_recovery (
  recovery_id uuid PRIMARY KEY,
  original_message_id text NOT NULL,
  source_destination text NOT NULL,
  failure_class text NOT NULL,
  attempt_count integer NOT NULL,
  first_failed_at timestamptz NOT NULL,
  last_failed_at timestamptz NOT NULL,
  traceparent text,
  redacted_payload jsonb NOT NULL,
  status text NOT NULL CHECK (status IN
    ('pending', 'approved', 'replayed', 'discarded')),
  approval_actor text,
  remediation_ref text,
  UNIQUE (source_destination, original_message_id, failure_class)
);

The unique constraint absorbs duplicate terminal handling. Keep immutable approval, replay, and discard events.

Java Spring Boot failure handler

This is application-managed publish-confirm-then-ack, not built-in DLX. publishConfirmed means confirmed routed durable acceptance. A RabbitMQ/Spring AMQP adapter must correlate each publish, require a positive publisher confirm, enable publisher returns, and publish with mandatory=true (or an equivalent routing check). A negative confirm, timeout, or returned/unroutable message rejects the call and leaves the source unacknowledged. A local RabbitTemplate return or exchange confirm alone is insufficient.

@Component
public class InventoryListener {
    private static final int MAX_ATTEMPTS = 3;
    private final InventoryService inventory;
    private final RecoveryPublisher publisher;
    private final FailureClassifier classifier;

    public InventoryListener(
        InventoryService inventory,
        RecoveryPublisher publisher,
        FailureClassifier classifier
    ) {
        this.inventory = inventory;
        this.publisher = publisher;
        this.classifier = classifier;
    }

    public void onMessage(
        OrderPlaced event,
        Delivery delivery,
        Channel channel
    ) throws Exception {
        try {
            inventory.reserveOnce(event); // PostgreSQL transaction committed
        } catch (Exception error) {
            Failure failure = classifier.classify(error);
            int nextAttempt = delivery.attempt() + 1;

            if (failure.transientFailure() && nextAttempt < MAX_ATTEMPTS) {
                publisher.publishConfirmed(
                    "inventory.retry",
                    RecoveryEnvelope.retryOf(event, delivery, failure, nextAttempt)
                        .redacted()
                );
                channel.basicAck(delivery.tag(), false);
                return;
            }

            publisher.publishConfirmed(
                "inventory.recovery",
                RecoveryEnvelope.terminal(event, delivery, failure, nextAttempt)
                    .redacted()
            );
            channel.basicAck(delivery.tag(), false);
            return;
        }

        // An acknowledgment error escapes as delivery ambiguity.
        channel.basicAck(delivery.tag(), false);
    }
}

If classification, redaction, or confirmed publication throws, the method exits before basicAck; RabbitMQ can requeue the unacknowledged delivery when the channel closes. A real adapter may explicitly negative-ack. Never acknowledge in finally.

reserveOnce must insert (consumer_name, message_id) into processed_message and update inventory in the same PostgreSQL transaction. A replay of msg-847 therefore cannot reserve twice. The retry route needs delayed delivery rather than immediate republish to avoid a hot loop.

Node.js TypeScript failure handler

The Node.js example uses the same strong contract. Where the provider supports routed acceptance, sendConfirmed must correlate its confirm with the publish and also await the provider’s unroutable-return result. For RabbitMQ it uses a confirm channel plus mandatory: true and a correlated return listener. It rejects on negative/uncertain confirmation or return; it must not silently weaken the contract to “exchange accepted.” delivery.ack() runs only after the promise resolves.

async function handleOrderPlaced(
  delivery: Delivery<OrderPlaced>,
  broker: ConfirmingBroker,
  inventory: InventoryService,
): Promise<void> {
  try {
    await inventory.reserveOnce(delivery.body); // committed or duplicate no-op
  } catch (error) {
    const failure = classifyFailure(error);
    const nextAttempt = delivery.attempt + 1;
    const envelope = redactEnvelope({
      originalMessageId: delivery.body.messageId,
      sourceDestination: delivery.source,
      failureClass: failure.kind,
      attemptCount: nextAttempt,
      firstFailedAt: delivery.firstFailedAt ?? new Date().toISOString(),
      lastFailedAt: new Date().toISOString(),
      traceparent: delivery.traceparent,
      payload: delivery.body,
    });

    const destination =
      failure.retryable && nextAttempt < 3
        ? "inventory.retry"
        : "inventory.recovery";

    await broker.sendConfirmed(destination, envelope);
    await delivery.ack();
    return;
  }

  // An acknowledgment error escapes as delivery ambiguity.
  await delivery.ack();
}

If sendConfirmed rejects or its result is ambiguous, ack() is never called. Redelivery can publish a duplicate envelope, so use the original message ID plus source and failure class as the recovery idempotency key. If ack() itself is lost after success, the same rule absorbs the repeat.

Inspection, remediation, and approved replay

A recovery console should support a deliberate workflow, not a “replay all” button:

Inspect: validate schema, failure history, message age, trace, business state, and whether the original effect committed.
Remediate: deploy code, repair reference data, restore a dependency, or create an explicit business correction. Do not edit evidence in place.
Approve: record actor, scope, reason, remediation reference, destination, rate, expiry, and a query selecting exact recovery IDs. High-impact payment replay should require separation of duties.
Replay: start with a canary batch, rate-limit it, preserve the original message ID or set causation_id to it, and run the normal idempotent handler.
Verify: compare replayed, duplicate, failed, and business-effect counts before widening the batch.
Discard: record an immutable reason and actor; never silently delete.

The owning domain team decides remediation. Platform messaging owns broker health and tooling, while security or compliance approves restricted payload access. Manual discard is acceptable only when the message is provably obsolete, superseded, an unauthorized test, irrecoverably malformed with no lawful repair, or already satisfied by verified business state. Never discard merely to clear an alert.

Blind automatic replay can recreate a poison loop, overwhelm a dependency, or apply obsolete business rules. Require human or tightly governed authorization.

Poison messages and malformed payloads

Validate the envelope before business deserialization: content type, size, schema version, stable message ID, event type, and required fields. Parse with limits. A malformed payload goes directly to restricted recovery after at most one diagnostic attempt; repeatedly parsing the same bytes cannot repair them.

A poison message may be valid yet trigger deterministic failure. Track payload hash and failure fingerprint. If many IDs share one fingerprint, pause that event version while healthy work continues where ordering permits.

Do not “fix” poison data by editing it and reusing the same identity without an audit trail. A transformed repair message needs a new message ID and an explicit causation_id pointing to the original. Ordinary replay of unchanged work retains the original identity.

Broker-specific behavior

RabbitMQ-specific triggers and routing: RabbitMQ dead-letters after rejection without requeue, message TTL expiry, queue-length eviction, or a quorum queue delivery-limit breach. It republishes to the configured dead-letter exchange, optionally with a different routing key. A DLX is a normal exchange, not universally a queue. Prefer policies over hard-coded queue arguments.

RabbitMQ-specific default safety: built-in DLX is separate from the application-managed publishConfirmed path above. By default, RabbitMQ’s internal DLX republish uses no publisher confirms and removes the message from the source immediately after publishing. If the target queue is unavailable to accept it, the dead-letter can be lost; a missing DLX can also silently drop it. This default is not confirmed routed durable acceptance.

RabbitMQ-specific quorum option: at-least-once dead-lettering is an opt-in for a source quorum queue; at-most-once remains its default. Set policy keys dead-letter-strategy=at-least-once, overflow=reject-publish, and dead-letter-exchange, and ensure the stream_queue feature flag is enabled. RabbitMQ then retains the dead-letter at the source while its internal consumer publishes with confirms and acknowledges only after target acceptance. Configure source length limits, durable targets, and persistent original messages. Missing or unroutable targets are retried and can create duplicates, so consumers still need idempotency.

Amazon SQS-specific: a source queue’s redrive policy names a DLQ and maxReceiveCount. SQS moves a repeatedly received but undeleted message after that threshold. The DLQ must be in the same account and Region, and its retention should exceed the source queue’s retention. A redrive allow policy controls which source queues may use it.

Amazon SQS-specific replay: StartMessageMoveTask can redrive messages to a source or compatible custom destination at a controlled velocity. AWS states that redrive assigns a new SQS messageID and enqueue time, so the application envelope must retain its original stable ID. SQS redrive cannot filter or modify messages; select or transform messages in an audited application workflow when remediation requires either. Redrive can also interleave recovered and new traffic, so FIFO users must analyze ordering impact.

Provider delivery counts are signals, not complete business attempt ledgers. Console inspection, visibility timeouts, requeues, and adapter behavior can affect counts. Preserve application attempt metadata independently.

Testing strategy

Test the whole terminal path, not only exception classification:

Exhaust a transient retry budget and assert exactly one logical recovery record with the original message ID.
Deliver malformed JSON and a valid poison message; assert neither enters an infinite retry loop and both store redacted diagnostics.
Simulate confirmed recovery publication followed by a lost source acknowledgment; assert redelivery deduplicates the recovery marker.
Simulate uncertain recovery publication; assert the source is not acknowledged.
Approve replay, run the original message twice, and assert one inventory reservation through the real PostgreSQL unique constraint.
Assert unapproved, expired, or over-limit replay requests are rejected.
Seed API keys, authorization headers, card-like values, and long stack traces; assert none appear in stored envelopes, logs, or alerts.
Replay a canary through inventory, payment, and notification test doubles with stable IDs and verify settlement occurs after each durable effect.

Use broker integration tests for headers, routing, visibility, confirms, and redrive. Mocks cannot prove settlement or PostgreSQL concurrency.

Monitoring and operations

Page on recovery age, growth, failure rate, and business impact, not queue depth alone. Track oldest pending age, arrivals and net growth per minute, retry-exhaustion rate, repeated failure fingerprints, replay success and re-failure, and discard volume. Add business dimensions such as orders awaiting inventory, payment value at risk, notification deadline breaches, and affected tenants.

Each destination needs an owner, severity rules, runbook, access policy, retention, and response objective. Page immediately for payment policy failures or redaction breaches; use sustained thresholds for notifications. Watch backlog and saturation during replay. Audit every operator action.

Common mistakes

Treating the DLQ as infinite storage with no owner or retention.
Retrying permanent, poison, malformed, expired, or policy failures.
Using unbounded immediate retries and creating a hot loop.
Acknowledging the source before durable retry or recovery acceptance.
Copying secrets and full payloads into recovery metadata.
Generating a new application message ID for ordinary replay.
Replaying around the normal idempotent consumer.
Bulk redriving without remediation, approval, canaries, or rate limits.
Alerting only on count while old high-value orders remain stuck.
Assuming RabbitMQ DLXs or Amazon SQS redrive semantics are universal.
Discarding records just to make the dashboard green.

Practical checklist

Classify transient, permanent, poison, malformed, expired, and policy failures.
Bound retry attempts and elapsed time; add backoff, jitter, and deadlines.
Preserve original message ID, source, class, attempts, times, and trace context.
Redact with an allowlist before durable recovery storage.
Confirm retry or recovery acceptance before source settlement.
Use a PostgreSQL unique constraint as a durable recovery marker.
Keep business mutation and processed-message marker in one transaction.
Assign a domain owner, runbook, retention, and manual discard criteria.
Alert by age, growth, failure fingerprint, and business impact.
Require recorded remediation and approval before replay.
Canary and rate-limit replay while monitoring source health.
Preserve identity or explicit causation and always pass through idempotency.

Frequently asked questions

Should every failed message go directly to a DLQ?

No. Retry transient failures within a bounded budget. Send deterministic, expired, policy, malformed, and exhausted failures to operated recovery.

Should replay create a new message ID?

Ordinary replay should retain the original application ID. If a provider creates a new transport ID, carry the original in the envelope. A deliberately transformed repair command gets a new ID plus causation_id.

Can a DLQ guarantee exactly-once recovery?

No. Publication and acknowledgment can be ambiguous. Duplicate-safe recovery comes from stable identity, durable uniqueness, and an idempotent business boundary.

When may an operator discard a message?

Only with recorded evidence that it is obsolete, superseded, unlawful to process, irrecoverably malformed, a test artifact, or already satisfied. The domain owner must approve according to impact.

Sources

Continue through the System Design learning path and browse topic clusters.
Set bounded attempts with Retry Patterns Explained.
Understand acknowledgment ambiguity in Message Delivery Semantics Explained.
Make replay safe with Idempotent Consumer Pattern Explained.
Build the upstream publication boundary with Transactional Outbox Pattern Explained.