When your service changes business data and also needs to trigger background work, you have to keep two systems in sync: the database that owns the data and the queue or broker that delivers the work. A user places an order, your code inserts the order row, and somewhere in the same request you need to make sure another system later charges the card, sends the email, or updates a search index. The database and the broker are separate systems, and no single statement can commit to both.
This is the dual-write problem. If you publish the work after the database commit, a crash in between loses the work while the database keeps the change. If you publish before the commit, you announce work that the transaction might roll back. Either way the two systems drift apart.
The transactional outbox pattern solves this by having your service write the business change and the job record into a database table called the outbox in one transaction. A separate process, the message relay, later reads those records and delivers them. Because the job record commits with the business data, the database never commits a business change without a matching job record. If anything crashes before delivery, the record survives and the relay retries.
By the end you will know how to save background work in the same transaction as the business change, recover it after a crash, make the consumer safe when a job runs twice, and keep adding a job to the queue separate from the work the job does.
Why the database write and the message send cannot be one transaction
The naive approach is to send the message from the code that updates the database. Both possible orders fail.
If you send after the commit, you are betting the process stays alive between the commit and the send. Sending after the commit risks a crash before the message is sent, which loses the message while the database keeps the change. Nothing retries it, because the code path that would have sent it is gone.
If you send before the commit, the failure is worse. The broker carries the message, then the database transaction rolls back, and a downstream service consumes an OrderCreated event for an order that was never committed. It may charge a customer or reserve inventory based on an order that does not exist.
Two-phase commit (2PC) is the textbook mechanism for making two systems commit together: a coordinator asks each participant to prepare, then tells them all to commit or roll back. It is rarely practical here. The database and the broker might not support it, and 2PC couples the service to both systems in a way that makes the broker a second transactional participant. The outbox pattern exists partly to avoid turning the database into a distributed transaction coordinator.
How the outbox pattern works
The outbox pattern moves the message send into the database. The service writes the business rows and an outbox record in the same local transaction. The outbox table acts as temporary storage for messages. Once the transaction commits, both the business change and the pending job record are durable. A separate process then reads the outbox and delivers each record to the broker or job runner.
A concrete example writes the order and the event in one transaction:
BEGIN TRANSACTION;
INSERT INTO orders (id, details) VALUES (1, 'Order Details');
INSERT INTO outbox (id, payload, status)
VALUES (1, '{"event": "OrderCreated"}', 'PENDING');
COMMIT;
The participants are the sender service, the database, the outbox table, and the relay. The sender writes the record, the database stores it, and the relay reads it and publishes it.
The message is sent if and only if the database transaction commits. The relay sends records in the order the service created them. And the code that writes business data never touches the broker, so the broker can be down without losing jobs: records queue up in the outbox table and go out when the broker recovers. You can swap the broker without changing the code that writes business data.
The outbox also separates adding a job to the queue from the work the job does. The enqueue step is the outbox insert inside your business transaction, the act of saying "this job needs to happen". The side effect is the actual work the job performs when it runs: sending the email, charging the card, updating the search index. Between the two sits the relay, and that gap is what makes retries possible. If the side effect fails, you do not lose the job record, you retry it.
Designing the outbox table and job records
The outbox table is the queue, so its columns need to carry everything the relay needs to deliver the job and everything you need to debug it later. A practical schema in Postgres looks like this:
CREATE TABLE outbox (
id UUID PRIMARY KEY,
aggregate_type VARCHAR(100) NOT NULL,
aggregate_id VARCHAR(100) NOT NULL,
event_type VARCHAR(200) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMP NOT NULL,
published_at TIMESTAMP NULL
);
The aggregate columns identify the business object the job belongs to, such as an order or a customer. event_type names the job, and payload carries the data the consumer (the system that runs the job) needs. published_at being NULL marks the record as not yet delivered. A simpler variant uses a status column with created_at, processed_at, and a trace_id so you can correlate the job with the request that created it.
For a retryable background job, the payload should contain the job description: the action to run and its arguments, plus a stable message ID the consumer can use to recognize duplicates. Store the complete payload at the time of the transaction. If you store only metadata and reconstruct the payload later, the underlying data may have changed since the event was created, and you can publish state that no longer matches what the transaction actually did.
The main way this pattern fails in practice is the developer who updates the business rows and forgets the outbox insert. Treat the two inserts as one unit in code: same transaction, same function, same code review.
Building the relay
The relay is the process that turns outbox rows into delivered work. There are two common implementations: polling the table, and tailing the database transaction log.
The polling publisher runs a query on an interval, grabs unpublished rows, sends them, and marks them published. A typical query is:
SELECT * FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100;
The dispatch itself is a simple loop: select the PENDING rows, send each payload to the broker, then update the row to PUBLISHED.
The polling interval is a latency and load trade-off. A 100 ms interval delivers jobs quickly but runs the query ten times a second; a 1 second interval cuts the query load and adds up to a second of latency. For low-volume workloads, polling is simple and often sufficient.
Batch size works the same way. Larger batches reduce the number of queries, network round trips, and broker calls, but a huge batch creates memory pressure, a long processing time, and a large retry window if the relay crashes partway.
If you run more than one relay instance, a plain SELECT with LIMIT lets two workers claim the same rows. Lock the rows you take and skip rows another worker already locked:
SELECT * FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED;
One boundary matters more than the rest: do not hold a database transaction open while you wait for the broker. The relay should read the records, publish outside any open transaction, and only then update the status. If you hold the transaction open across the broker call, you create long transactions, lock contention, and connection exhaustion, and the database ends up waiting on an external system.
Instead of polling, you can use transaction log tailing, also called change data capture (CDC). Tools such as Debezium read the database’s write-ahead log, the append-only record of committed changes, so they observe new outbox rows as they are committed instead of repeatedly scanning the table. CDC offers the highest throughput, low latency, and strong ordering, at the cost of operational complexity. Start with polling unless you already run CDC infrastructure.
Failure cases: what the outbox recovers and what it does not
The outbox changes what happens when things crash, but it does not make crashes invisible.
The database commits and the service crashes before the relay publishes. The job record is already in the outbox table. When the relay restarts, it reads the record and publishes it, so the job is not lost.
The case it does not fix: the relay publishes the message, then crashes before marking the row PUBLISHED. On restart it reads the same row and publishes again. The same thing happens if the broker accepts the message but the acknowledgment is lost. The practical delivery guarantee is therefore at-least-once: if the transaction commits, the job will eventually be delivered, but it may be delivered more than once.
No relay design eliminates that window. Any scheme that marks the row after publishing has a gap in which the publish happened but the mark did not. Consumers have to tolerate duplicates.
Broker downtime fits the same model. If the broker is unreachable, the relay keeps failing and the records stay PENDING. They queue up in the outbox and go out when the broker recovers, so the backlog shows up in monitoring rather than as lost jobs.
On the producing side, the relay sends records in the order the service created them, and for jobs that update the same business object you want that order kept across service instances. That ordering does not automatically carry through to the consumer. If order matters to the job, the broker and the consumer must preserve it explicitly.
Making the consumer safe against duplicates
Because delivery is at-least-once, the worker that runs the job must be idempotent: running the same job twice must not produce two side effects.
The idempotent consumer pattern records each processed message ID in a database table, checks it before handling a message, and commits the side effects and that record in the same transaction. A redelivered duplicate then finds the ID, short-circuits, and produces no second side effect.
A unique constraint on (MessageId, ConsumerName) in the processed-messages table prevents race conditions. If two copies of the same message arrive concurrently, only one insert succeeds; the other fails the constraint and skips the work.
Broker-side deduplication, such as the duplicate detection that Azure Service Bus and Amazon SQS offer, solves a different problem. It catches duplicates from producer-side network retries within a configured window, but it does not help when the broker redelivers a message because the consumer did not acknowledge it, or when the consumer crashes mid-processing. You still need the consumer-side check.
Some jobs are naturally idempotent. A consumer that sets a status flag, updates a projection (a read model derived from the source data), or refreshes a cache produces the same result on the second run, so a precondition check that returns early when the entity is already in the desired state is often enough.
External side effects such as sending email or charging a card sit outside your transaction boundary, and they are the hard case. If the network connection drops after the external service receives the request, the consumer cannot tell whether the charge happened, and a naive retry can charge twice. Two strategies work. If the external service supports idempotency keys, pass a stable key derived from the message ID so the provider recognizes a retry. Otherwise, store a record of the intended action in your own database and let a background process perform that action once, guarded by the same processed-record table.
The inbox pattern is the mirror image of the outbox on the consumer side. Upon receiving a message, the consumer stores it in an inbox table in the same transaction that handles the initial business data, checks the inbox before processing, and acknowledges and ignores duplicates. It gives you the same at-least-once safety when the consumer owns a database.
Retrying failed jobs
A job can fail after the relay delivers it, or the relay can fail to publish it. Both cases need a retry path, and the outbox table carries the state for both.
The status column drives the retries. PENDING means not yet claimed, PROCESSING means a worker has locked the row, PUBLISHED is terminal, and FAILED branches off PROCESSING. When the relay fails to publish a row and the failure is retryable, the row returns to PENDING so the next poll picks it up again. After the retry count passes a threshold you configure, the row moves to DEAD_LETTER and stays there for manual inspection. The exact retry schedule, fixed delay or exponential backoff, is a configuration choice; the table just needs the retry count and the state.
The outbox grows continuously and needs a retention strategy. You can delete published rows after a retention period, for example DELETE FROM outbox WHERE published_at < now() - interval '7 days', or partition by date so old partitions drop quickly. The right choice depends on whether you need the records for audit, replay, or compliance, and on how large the table gets at your event volume.
You also need to watch the outbox so you know when the relay is falling behind. Track the outbox size, the oldest unpublished event, and publishing latency. Also track publishing failures, retry counts, and dead-letter events. The most useful single number is outbox lag: the time between now and the oldest unpublished record. When the relay is healthy, lag stays near the polling interval. When lag grows, the relay is stuck or the broker is down.
Every business transaction now carries an extra write, which increases transaction size and contention on the outbox table. Delivery becomes asynchronous, so jobs arrive a polling interval later. You operate a relay, a retention policy, and monitoring. And the delivery guarantee is at-least-once, so duplicates are a normal part of failure handling. For a system where losing a background job means losing money or trust, that trade is usually worth it.
