Transactional Outbox Pattern Explained

Quick answer

The transactional outbox pattern prevents a service from committing business data while losing the event that describes that change. In one local database transaction, the service updates its aggregate and inserts an immutable publication-intent row into an outbox table. A separate publisher later locks a bounded batch of rows, sends them to the broker, and records successful publication.

This removes the database-and-broker dual write from the request path. It does not make delivery exactly once. If the broker accepts an event and the publisher crashes before setting published_at, that row will be retried and published twice. Treat the relay as at-least-once delivery: use a stable event ID, send it as broker metadata, and require every consumer to be idempotent.

Use the pattern when one service owns the business table and outbox in the same transactional database. It is not a cross-database atomicity mechanism and it does not replace clear event contracts, retry policy, monitoring, or retention.

Failure timeline

Consider an order service that must persist an order and publish OrderPlaced.

The request starts a PostgreSQL transaction.
It inserts the order.
It inserts an outbox row with event ID f81d....
The transaction commits both rows, or rolls back both.
A publisher selects that row with FOR UPDATE SKIP LOCKED.
It sends the event with f81d... as the message ID.
It updates the row as published and commits.

Three interruption points explain the guarantee:

A crash before step 4 leaves neither order nor publication intent.
A crash after step 4 but before step 6 leaves a durable pending row for another attempt.
A crash after broker acceptance in step 6 but before the commit in step 7 leaves the row pending. The retry can create a duplicate, so the consumer must deduplicate f81d....

That first event starts a larger failure chain without making the order service responsible for every step. An inventory service consumes OrderPlaced, reserves stock, and emits InventoryReserved; a payment service then collects payment and emits PaymentCollected; a notification service finally sends the receipt. A lost OrderPlaced leaves inventory, payment, and notification untouched even though the order exists. A duplicate can reserve twice, charge twice, or send repeated messages if downstream consumers are not idempotent. Each service should atomically store its own state change and next outbox event, carrying stable message identity and correlation data through the chain.

The design closes the “committed but never announced” gap. It deliberately accepts a detectable duplicate window instead of pretending two independent systems share one atomic commit.

The dual-write problem

A database transaction cannot normally roll back a message already accepted by an independent broker. Sending first is unsafe: the broker may deliver OrderPlaced, then the database transaction may fail. Committing first is also unsafe: the service may crash after the commit but before the send. Retrying the HTTP request does not prove which side succeeded and may add more duplicates.

Two-phase commit can coordinate some resource managers, but it increases coupling and is often unsupported or intentionally avoided. The named Transactional Outbox pattern changes the problem: the request writes only to its authoritative database, while a relay performs publication later. PostgreSQL supplies the local atomicity; the broker remains outside that transaction.

The outbox invariant

The useful invariant is:

A committed business change that requires an event has exactly one durable outbox intent in the same local transaction; a rolled-back change has none.

“Exactly one intent” is not “exactly one delivery.” Generate the event ID once in application code and insert it with the aggregate mutation. Do not let the relay invent a new ID on every attempt. Keep the payload and event type immutable after commit; a correction should be a new event.

The invariant must be enforced at every write path. A repository method that can update an order without inserting its required event is an escape hatch. Put the aggregate update and outbox insert behind one transactional application service, add integration tests, and restrict direct writes where practical.

PostgreSQL schema and transaction

This schema records identity, immutable content, publication state, and retry scheduling:

CREATE TABLE outbox_event (
    event_id         uuid PRIMARY KEY,
    aggregate_type   text NOT NULL,
    aggregate_id     text NOT NULL,
    event_type       text NOT NULL,
    payload          jsonb NOT NULL,
    created_at       timestamptz NOT NULL DEFAULT clock_timestamp(),
    status           text NOT NULL DEFAULT 'PENDING'
                     CHECK (status IN ('PENDING', 'PUBLISHED', 'FAILED')),
    attempt_count    integer NOT NULL DEFAULT 0 CHECK (attempt_count >= 0),
    next_attempt_at  timestamptz NOT NULL DEFAULT clock_timestamp(),
    published_at     timestamptz,
    last_error       text,
    CHECK (
      (status = 'PUBLISHED' AND published_at IS NOT NULL)
      OR (status <> 'PUBLISHED' AND published_at IS NULL)
    )
);

CREATE INDEX outbox_event_ready_idx
    ON outbox_event (next_attempt_at, created_at, event_id)
    WHERE status = 'PENDING';

The partial index supports the publisher’s ready-row scan. Keep last_error bounded in application code so an exception cannot turn the table into a log archive. A UUID generated by the application is stable across retries and is the primary key.

The business transaction is auditable SQL:

BEGIN;

INSERT INTO customer_order (order_id, customer_id, total_cents, status)
VALUES ('ord-1042', 'cus-7', 4599, 'PLACED');

INSERT INTO outbox_event (
    event_id, aggregate_type, aggregate_id, event_type, payload
) VALUES (
    'f81d4fae-7dec-4e4c-a765-00a0c91e6bf6',
    'Order',
    'ord-1042',
    'OrderPlaced.v1',
    '{"orderId":"ord-1042","customerId":"cus-7","totalCents":4599}'::jsonb
);

COMMIT;

PostgreSQL transactions make the two inserts an all-or-nothing unit. The official transactions tutorial explains that intermediate changes remain invisible until commit and are discarded by rollback.

Java Spring Boot implementation

Use one Spring-managed transaction for the business row and outbox row. Serialize before the first database write so serialization failure cannot leave partial work:

@Service
public class OrderApplicationService {
    private final JdbcTemplate jdbc;
    private final ObjectMapper json;

    public OrderApplicationService(JdbcTemplate jdbc, ObjectMapper json) {
        this.jdbc = jdbc;
        this.json = json;
    }

    @Transactional
    public UUID placeOrder(Order order) {
        UUID eventId = UUID.randomUUID();
        String payload;
        try {
            payload = json.writeValueAsString(new OrderPlaced(
                order.id(), order.customerId(), order.totalCents()));
        } catch (JsonProcessingException error) {
            throw new IllegalArgumentException("Cannot serialize OrderPlaced", error);
        }

        jdbc.update("""
            INSERT INTO customer_order
                (order_id, customer_id, total_cents, status)
            VALUES (?, ?, ?, 'PLACED')
            """, order.id(), order.customerId(), order.totalCents());

        jdbc.update("""
            INSERT INTO outbox_event
                (event_id, aggregate_type, aggregate_id, event_type, payload)
            VALUES (?, 'Order', ?, 'OrderPlaced.v1', ?::jsonb)
            """, eventId, order.id(), payload);
        return eventId;
    }
}

Spring’s @Transactional documentation documents the default PROPAGATION_REQUIRED behavior and rollback rules. Keep the method public and invoke it through the Spring bean; proxy-based transaction management does not intercept ordinary self-invocation. Spring’s JDBC guide documents JdbcTemplate query and update behavior.

The EventBroker abstraction used below has a strict completion contract: publish returns normally, or its promise resolves, only after the broker confirms acceptance at the durability level the service has chosen. Putting a message into a local client buffer, in-memory queue, or framework executor is insufficient. Kafka acknowledgment configuration, RabbitMQ publisher confirms, and framework-specific send futures are possible implementations, but their guarantees and settings are broker-specific. The adapter must wait for the applicable confirmation and surface rejection, timeout, or ambiguous completion as a failure.

A small polling publisher can hold row locks while it calls that confirmed broker adapter:

@Service
public class OutboxPublisher {
    private static final int BATCH_SIZE = 50;
    private final JdbcTemplate jdbc;
    private final EventBroker broker;

    public OutboxPublisher(JdbcTemplate jdbc, EventBroker broker) {
        this.jdbc = jdbc;
        this.broker = broker;
    }

    @Transactional
    public int publishBatch() {
        List<OutboxEvent> events = jdbc.query("""
            SELECT event_id, event_type, payload::text
            FROM outbox_event
            WHERE status = 'PENDING' AND next_attempt_at <= clock_timestamp()
            ORDER BY created_at, event_id
            LIMIT ?
            FOR UPDATE SKIP LOCKED
            """, outboxRowMapper(), BATCH_SIZE);

        for (OutboxEvent event : events) {
            try {
                broker.publish(event.eventType(), event.eventId().toString(),
                               event.payload());
                jdbc.update("""
                    UPDATE outbox_event
                    SET status = 'PUBLISHED', published_at = clock_timestamp(),
                        attempt_count = attempt_count + 1, last_error = NULL
                    WHERE event_id = ?
                    """, event.eventId());
            } catch (RuntimeException error) {
                jdbc.update("""
                    UPDATE outbox_event
                    SET attempt_count = attempt_count + 1,
                        next_attempt_at = clock_timestamp() + interval '30 seconds',
                        last_error = left(?, 1000)
                    WHERE event_id = ?
                    """, error.getMessage(), event.eventId());
            }
        }
        return events.size();
    }
}

Configure broker timeouts shorter than the scheduler interval and keep the batch bounded. A slow broker otherwise holds locks and a database connection for too long.

Node.js TypeScript implementation

With node-postgres, every statement in a transaction must use the same checked-out client. The official node-postgres transaction guide warns against mixing pool.query with a client transaction.

import { randomUUID } from "node:crypto";
import type { Pool, PoolClient } from "pg";

export async function placeOrder(pool: Pool, order: Order): Promise<string> {
  const client = await pool.connect();
  const eventId = randomUUID();
  try {
    await client.query("BEGIN");
    await client.query(
      `INSERT INTO customer_order
         (order_id, customer_id, total_cents, status)
       VALUES ($1, $2, $3, 'PLACED')`,
      [order.id, order.customerId, order.totalCents],
    );
    await client.query(
      `INSERT INTO outbox_event
         (event_id, aggregate_type, aggregate_id, event_type, payload)
       VALUES ($1, 'Order', $2, 'OrderPlaced.v1', $3::jsonb)`,
      [eventId, order.id, JSON.stringify(order)],
    );
    await client.query("COMMIT");
    return eventId;
  } catch (error) {
    await client.query("ROLLBACK");
    throw error;
  } finally {
    client.release();
  }
}

The publisher uses parameterized statements and the same client until commit:

type OutboxRow = {
  event_id: string;
  event_type: string;
  payload: unknown;
};

export async function publishBatch(
  pool: Pool,
  broker: EventBroker,
  batchSize = 50,
): Promise<number> {
  const client: PoolClient = await pool.connect();
  try {
    await client.query("BEGIN");
    const result = await client.query<OutboxRow>(
      `SELECT event_id, event_type, payload
       FROM outbox_event
       WHERE status = 'PENDING' AND next_attempt_at <= clock_timestamp()
       ORDER BY created_at, event_id
       LIMIT $1
       FOR UPDATE SKIP LOCKED`,
      [Math.min(Math.max(batchSize, 1), 100)],
    );

    for (const event of result.rows) {
      try {
        await broker.publish({
          topic: event.event_type,
          messageId: event.event_id,
          payload: event.payload,
        });
        await client.query(
          `UPDATE outbox_event
           SET status = 'PUBLISHED', published_at = clock_timestamp(),
               attempt_count = attempt_count + 1, last_error = NULL
           WHERE event_id = $1`,
          [event.event_id],
        );
      } catch (error) {
        const message = error instanceof Error ? error.message : "publish failed";
        await client.query(
          `UPDATE outbox_event
           SET attempt_count = attempt_count + 1,
               next_attempt_at = clock_timestamp() + interval '30 seconds',
               last_error = left($2, 1000)
           WHERE event_id = $1`,
          [event.event_id, message],
        );
      }
    }

    await client.query("COMMIT");
    return result.rowCount ?? 0;
  } catch (error) {
    await client.query("ROLLBACK");
    throw error;
  } finally {
    client.release();
  }
}

The fixed retry shown here is readable, not ideal policy. Production code should use capped exponential backoff with jitter and a maximum-attempt transition.

As in the Java example, await broker.publish(...) must await broker-confirmed acceptance, not merely local enqueue. Confirmation establishes the send side of the handoff; it does not atomically update PostgreSQL. A process can still die after confirmed acceptance but before COMMIT, preserving the required duplicate-publication window.

Publishing with competing workers

FOR UPDATE locks selected rows until transaction end. SKIP LOCKED makes another worker skip rows already claimed rather than wait behind them. PostgreSQL’s SELECT documentation explicitly says this gives an inconsistent view unsuitable for general reads but can avoid contention for multiple consumers of a queue-like table. That trade is correct here: each worker needs available work, not a consistent report of the whole outbox.

Always combine locking with deterministic ORDER BY and LIMIT. The limit bounds broker calls, lock duration, and recovery work. Multiple workers increase throughput, but ordering is subtle: locks prevent concurrent ownership of one row, not global broker ordering. If per-aggregate order is required, partition by aggregate key or publish through one ordered lane per key and include an aggregate sequence.

Crash windows and duplicate publication

No order of the broker send and database update removes every crash window. Marking published before sending can lose an event. Sending before marking can duplicate it. The outbox chooses the second failure mode because duplicates can be detected by stable identity, while an unrecorded loss is difficult to recover.

The critical case must be explicit: a crash after broker acceptance but before marking the row published creates a duplicate and therefore requires an idempotent consumer. The next guide, Message Delivery Semantics Explained, distinguishes these guarantees, and Idempotent Consumer Pattern Explained shows how to store processed message IDs with consumer-side effects.

Cleanup, retention, and poison events

Keep published rows long enough for incident investigation and replay decisions, then delete them in small batches. The outbox is delivery infrastructure, not necessarily the permanent business audit log.

WITH expired AS (
  SELECT event_id
  FROM outbox_event
  WHERE status = 'PUBLISHED'
    AND published_at < clock_timestamp() - interval '14 days'
  ORDER BY published_at
  LIMIT 1000
)
DELETE FROM outbox_event o
USING expired
WHERE o.event_id = expired.event_id;

After a configured attempt ceiling, set status = 'FAILED', retain the payload and last error, alert operators, and require an audited replay or discard decision. Do not let one poison event block later rows indefinitely. Validate payloads before insertion, but expect permanent broker rejections, revoked schemas, oversized messages, and bad routing. A dead-letter workflow is operational policy, not a substitute for diagnosing the producer defect.

Testing strategy

Test the invariant and crash boundaries, not just repository methods:

In an integration test with PostgreSQL, force the business insert to fail and verify no outbox row commits.
Force the outbox insert to fail and verify the aggregate write rolls back.
Run two publishers concurrently and prove each locked event is handled by only one active worker.
Make the broker fail before acceptance; verify attempts increase and next_attempt_at moves forward.
Simulate acceptance followed by process failure before the database update; verify redelivery uses the same event ID and the consumer produces one business effect.
Verify batch limits, deterministic ordering, poison-event transition, and retention deletion.
Test serialization and contract compatibility with representative payload fixtures.

H2 is useful for fast Spring Boot repository tests, but it is not proof of PostgreSQL locking semantics. Keep at least one PostgreSQL integration suite because SKIP LOCKED, JSONB, partial indexes, and concurrency behavior are database-specific.

Monitoring and operations

Measure the system as a pipeline. Track pending row count, age of the oldest ready row, publication latency from created_at to published_at, attempts by event type, failed-row count, batch duration, broker latency, and publisher transaction errors. The oldest-row age is often more actionable than queue depth because a quiet but stuck partition can have few rows.

Alert on sustained age, not a single polling interval. Log event ID, aggregate identity, event type, attempt number, and trace or request ID; never log sensitive payloads by default. Dashboard database connection use and lock wait time because workers hold connections across broker calls. Provide a runbook for pausing publishers, inspecting failed rows, replaying by stable ID, and confirming consumer idempotency before a bulk replay.

Common mistakes

Publishing inside the request transaction. A broker call still cannot share PostgreSQL’s local atomic commit and makes request latency depend on the broker.

Writing the outbox after commit. That recreates the original crash gap.

Generating a new ID per retry. Consumers cannot recognize duplicate delivery.

Selecting without row locks. Competing workers can publish the same pending row simultaneously.

Unbounded batches. Large transactions hold locks and connections, amplify retries, and slow shutdown.

Assuming SKIP LOCKED preserves global order. It favors available work; locked earlier rows can be overtaken.

Marking published before broker acceptance. A crash can cause permanent loss.

Treating “published” as “processed.” It records relay progress, not consumer success.

Deleting failures automatically. Poison events need visibility and an explicit operational decision.

Practical checklist

Update business state and insert the outbox row in one local transaction.
Generate one stable event ID before the insert and reuse it on every attempt.
Store aggregate identity, event type, immutable payload, creation time, status, attempts, and next-attempt time.
Index the pending ready-row scan.
Poll a bounded, deterministically ordered batch with FOR UPDATE SKIP LOCKED.
Send the event ID in broker metadata.
Record success only after broker acceptance.
Design and test idempotent consumers for duplicate delivery.
Cap retries, add backoff and jitter, and surface poison events.
Monitor oldest pending age, publication latency, failures, locks, and connection use.
Retain published rows for a documented period and delete in bounded batches.
Test real PostgreSQL concurrency and every crash window.

Frequently asked questions

Does the transactional outbox guarantee exactly-once delivery?

No. It makes publication intent atomic with the business change and normally provides at-least-once relay behavior. A consumer must deduplicate stable event IDs.

Why not update the outbox before sending?

If “published” commits first and the process crashes before the broker accepts the event, the relay will skip a message that was never sent.

Is `SKIP LOCKED` mandatory?

No, but it is a practical PostgreSQL mechanism for competing polling workers. A single publisher can use ordinary locking; change-data-capture relays use a different mechanism.

Should the publisher keep a database transaction open during the broker call?

The simple polling design shown here does so to preserve row ownership. Keep batches and timeouts small. A claim-and-lease design can shorten transactions, but it needs lease expiry, recovery, and the same duplicate handling.

Can the outbox payload be changed before retry?

Do not mutate a committed event. Fix the producer and emit a new versioned event or use an audited repair workflow.

Sources

Continue through the System Design learning path and browse topic clusters.
Compare guarantees in Message Delivery Semantics Explained.
Make duplicate handling durable with Idempotent Consumer Pattern Explained.
Plan terminal failures with Dead-Letter Queue Explained.