Message Delivery Semantics Lab: Crash, Retry, and Idempotency Examples

A delivery guarantee is useful only when you name the boundary it describes. A broker can deliver a record, accept an acknowledgement, retain an offset, and make an unacknowledged record available again. Your consumer can start processing, commit database state, call an HTTP API, send an email, or charge a payment. Those events do not become one atomic action merely because they appear in one handler method.

The Message Delivery Semantics Lab turns that distinction into a deterministic timeline. It does not connect to Kafka, a database, or a remote API. Every run is a small state machine: identical inputs produce identical ordered steps and counters.

Quick answer

ACK before durable work can lose work after a crash. ACK after durable work protects against that loss, but an after-effect/before-ACK crash can cause redelivery and duplicate business effects. A stable message ID can make repeated processing converge, while bounded retries expose an explicit DLQ or human-recovery boundary. None of those controls makes an arbitrary payment, email, or HTTP request end-to-end exactly once.

Learning objectives

• Predict loss, redelivery, duplicate effects, and retry exhaustion by placing broker acknowledgement around a durable business side effect.
• Use stable message identity, bounded retries, DLQ ownership, and reconciliation without claiming arbitrary external effects are exactly once.

Prerequisites

Read Message Delivery Semantics Explained for the vocabulary behind at-most-once and at-least-once delivery. The Idempotent Consumer Pattern explains how duplicate detection and a database update can share one local transaction. The Dead Letter Queue guide covers the operated recovery state reached when automation stops.

How to read the lab

The input model has five choices:

1. Delivery policy decides whether an unacknowledged failure may be made available again.
2. ACK timing places broker progress before processing or after the modeled business side effect.
3. Failure point selects a no-failure run, a persistent failure before the effect, a one-time crash after the effect but before ACK, or a crash after ACK.
4. Idempotency either leaves the effect unguarded or uses one stable message/event ID across attempts.
5. Retry limit is an integer from one through five. It counts redelivery attempts after the initial delivery and can never be infinite.

The output keeps three counters separate. Deliveries count broker attempts. Successful processing counts attempts that reached and completed the modeled effect-handling stage. Business side effects count effects actually applied. With stable idempotency, the second processing attempt can succeed while returning the already-recorded result, so processing count can be two while the business-effect count remains one.

The model treats a crash before the side effect as a persistent processing failure. This makes retry exhaustion visible. It treats the after-effect/before-ACK crash as a one-time process crash: after restart, the replay reaches ACK unless the finite policy prevents redelivery. That distinction is part of the teaching model, not a universal broker feature.

Scenario 1: ACK before processing

Open the lab and choose ACK before processing. The preset uses at-most-once delivery, acknowledges before the handler does business work, and injects a crash before the side effect.

The timeline is:

1. The broker delivers the message.
2. The consumer records the ACK.
3. The handler starts.
4. The handler crashes before the business effect.
5. Work is lost because no unacknowledged broker delivery remains.

The final ACK is real broker-progress evidence. It is not evidence that the database changed or the user received the requested outcome. A production validation query must compare acknowledged progress with durable business records, missing-outcome counts, and user-visible status. If acknowledgement must happen early, create a different durable recovery record before ACK and operate that record explicitly.

Scenario 2: Crash after side effect, before ACK

Load Crash after side effect, before ACK. The initial attempt applies the business effect, then crashes before acknowledgement. Under the at-least-once policy, the broker makes the same logical message available again. Without duplicate protection, the replay applies the effect a second time and then records ACK.

The important gap is not “Kafka duplicated my database row.” The first application already committed an effect, while broker progress remained unrecorded. Redelivery is a defensible response to that missing ACK. The consumer owns the rule that decides whether applying the same logical request again is safe.

For a Java/Spring consumer, that rule might be a unique message_id stored in the same local database transaction as the projection update:

@Transactional
public void apply(OrderAccepted event) {
    if (!processedMessageRepository.tryInsert(event.id())) {
        return;
    }
    orderProjectionRepository.applyAccepted(event.orderId(), event.version());
}

The uniqueness constraint and projection update must commit together. A marker committed before the projection can hide missing work. A marker committed afterward can race and allow repeated work. An external payment provider is not inside this database transaction; it needs its own idempotency key or reconciliation contract.

Scenario 3: At-least-once with stable idempotency key

Load At-least-once with stable idempotency key. The crash window is the same as scenario 2, so redelivery still occurs. This time both attempts carry the same event ID. The first attempt applies the effect. The replay finds the stored result and suppresses a second application before recording ACK.

The result shows two deliveries, two successful processing outcomes, and one applied business side effect. This is convergence, not disappearance of duplicates. Metrics should expose the difference:

• delivery attempts and redelivery rate;
• idempotency claims created, reused, and conflicted;
• business effects applied;
• acknowledgement or offset progress;
• unresolved user outcomes and reconciliation discrepancies.

Stable identity must describe the same logical operation. Generating a fresh UUID on every retry defeats duplicate recognition. Reusing one identifier for different payloads is also unsafe; store an operation fingerprint or invariant and reject conflicting reuse.

Scenario 4: Retry exhaustion

Load Retry exhaustion. The handler fails before the side effect on every attempt. With a retry limit of two, the state machine performs one initial delivery and two redeliveries. No business effect succeeds, no final ACK is recorded, and automated processing stops at a manual-recovery boundary.

Retry exhaustion does not prove the request is invalid, lost, or successfully completed. It proves only that this automated policy stopped. A production DLQ or parking-lot contract needs:

• a stable event ID and original payload reference;
• failure classification and attempt history;
• ownership, alerting, and response objective;
• authorization for inspection, repair, replay, or terminal rejection;
• duplicate-safe replay and ordering checks;
• an audit record of the final business disposition.

The reliable Spring Kafka consumer lesson compares blocking retries, non-blocking retry topics, acknowledgements, and dead-letter topics. Those are versioned Spring Kafka behaviors and configuration choices, not guarantees shared by every message system.

Why there is no exactly-once toggle

Kafka transactions can atomically coordinate supported Kafka reads, writes, and offsets within their documented scope. Spring Kafka can help applications use those capabilities. That scope does not automatically include an arbitrary HTTP service, payment processor, SMTP server, or unrelated database. The lab therefore does not offer a universal “exactly once” checkbox and does not generate production Spring configuration.

For external effects, use an explicit combination of stable operation identity, provider idempotency where supported, durable outcome recording, status queries, reconciliation, and manual resolution. Describe exactly which boundary is atomic and which boundary is merely retried or reconciled.

Production validation

Validate the chain with independent evidence:

1. Confirm producer intent committed, such as a business row and transactional outbox record.
2. Confirm broker acceptance using producer results and broker telemetry, not only an application log line.
3. Confirm consumer ownership, delivery attempts, ACK or offset progress, retry location, and DLQ disposition.
4. Confirm the local database update and idempotency record committed under the intended uniqueness and transaction boundary.
5. Confirm external outcomes through provider status or reconciliation rather than assuming the local handler response proves completion.
6. Reconcile user-visible business state with broker and consumer progress before closing an incident.

A useful failure drill pauses the consumer after the business commit but before ACK, restarts it, and verifies that the same event ID reappears while the business result remains single. Run that only in an isolated environment with synthetic effects; this browser lab performs no real messages or side effects.

Common mistakes

• Treating an ACK or committed offset as proof of a database or external outcome.
• Treating a committed database row as proof the broker observed progress.
• Creating a new message ID for every redelivery.
• Recording duplicate detection separately from the local business update.
• Retrying permanent validation failures without a finite limit.
• Replaying a DLQ at unlimited speed or without authorization and ordering checks.
• Calling Kafka or Spring transaction support end-to-end exactly once for arbitrary external effects.

Related reading

Continue with Event Ordering, Deduplication, and State Convergence when entity versions must reject stale arrivals. Use Production Kafka and Event-Driven Systems Troubleshooting to connect lag, retries, durable state, and user recovery. Return to the Event-Driven Systems course to review the complete publication and consumption chain.

Sources

• Apache Kafka Design — delivery, replication, consumer position, and transaction design; accessed 2026-08-18.
• Spring for Apache Kafka Reference — listener containers, transactions, retries, and recovery; accessed 2026-08-18.
• CloudEvents Specification — stable event context and identity vocabulary; accessed 2026-08-18.
• AsyncAPI document concepts — channels, operations, messages, and bindings as interface documentation; accessed 2026-08-18.