The Problem: Updating a Database and Publishing an Event
Imagine an Order Service.
A customer places an order:
POST /orders
The Order Service needs to do two things:
1. Save the order in the database
2. Publish an OrderCreated event
The implementation might initially look simple:
Create Order
|
+----> Database
|
+----> Message Broker
For example:
@Transactional
public Order createOrder(CreateOrderRequest request) {
Order order = orderRepository.save(
new Order(request)
);
eventPublisher.publish(
new OrderCreated(order.getId())
);
return order;
}
It looks reasonable.
But there is a serious distributed-systems problem hiding here.
The database and the message broker are two different systems.
The database transaction can guarantee atomicity for database operations.
The message broker has its own state and its own failure modes.
There is no ordinary local transaction that can guarantee:
Database update
+
Message publication
either both happen or neither happens.
This is called the dual-write problem.
And it is one of the most common reliability problems in event-driven microservices.
What Is the Dual-Write Problem?
A dual write occurs when one business operation needs to update two independent systems.
For example:
Order Service
|
+----> PostgreSQL
|
+----> Kafka
The application wants:
Database update ✓
Message publish ✓
But there are multiple failure points.
Consider:
Save Order
|
v
Database COMMIT
|
X
Application crashes
|
v
Publish OrderCreated
The database contains the order.
But the event was never published.
Now downstream services do not know that the order exists.
The system is inconsistent.
This is exactly the problem the Transactional Outbox Pattern is designed to address.
The First Naive Approach
Let's start with the obvious implementation.
BEGIN TRANSACTION
INSERT INTO orders (...)
COMMIT
publish(OrderCreated)
It seems safe because the database transaction completes before the event is published.
But consider:
INSERT INTO orders
|
v
COMMIT
|
v
CRASH
|
X
publish(OrderCreated)
The order exists.
The event does not.
The system has lost the notification.
This is a lost event problem.
What If We Publish First?
Maybe we can reverse the order:
publish(OrderCreated)
BEGIN TRANSACTION
INSERT INTO orders (...)
COMMIT
Now we have the opposite problem.
Suppose:
publish(OrderCreated)
|
v
Message Broker ✓
|
v
Database INSERT
|
X
Database transaction fails
The event says:
OrderCreated
but the order does not actually exist.
A downstream service might try to process an order that was never committed.
This is even worse.
So:
Database → Broker
has one failure window.
And:
Broker → Database
has another.
Neither gives us atomicity.
What About a Database Transaction?
You might try:
@Transactional
public void createOrder() {
saveOrder();
publishEvent();
}
But the transaction only controls resources that participate in that transaction.
For example:
Database Transaction
|
+---- orders table
|
+---- payments table
|
+---- inventory table
The message broker is outside that transaction:
Database Transaction
|
+---- Database
|
X
Message Broker
If the broker is not participating in the same distributed transaction, the database transaction cannot roll back a message that has already been published.
What About Two-Phase Commit?
The theoretical solution is a distributed transaction.
For example:
Coordinator
|
+---------+---------+
| |
v v
Database Message Broker
The coordinator could ask both participants:
Can you commit?
Then:
Commit
This is the idea behind Two-Phase Commit (2PC).
But 2PC introduces significant complexity and coupling, and many databases and messaging systems do not participate in a common distributed transaction. The Transactional Outbox Pattern avoids requiring such a distributed transaction.
Instead of trying to make:
Database + Message Broker
one transaction, we change the design.
The Transactional Outbox Pattern
The key idea is surprisingly simple:
Store the event in the same database transaction as the business data. Publish the event later.
Instead of:
Database
+
Message Broker
we do:
Database
|
+---- Business Data
|
+---- Outbox
Both are written in the same local database transaction.
For example:
BEGIN TRANSACTION
INSERT INTO orders (...)
INSERT INTO outbox (...)
COMMIT
Now there is only one transaction.
Either:
orders ✓
outbox ✓
or:
orders ✗
outbox ✗
The event has not yet been sent to the broker.
It has been stored safely in the database.
A separate process publishes it later.
This is the core of the Transactional Outbox Pattern.
The Architecture
The basic architecture becomes:
Order Service
|
v
Local Database
/ \
/ \
v v
Orders Table Outbox Table
|
|
Outbox Publisher
|
v
Message Broker
|
+-------------+-------------+
| | |
v v v
Payment Service Inventory Service Notification Service
The application no longer directly publishes the event as part of the request.
Instead:
Application
|
+----> Business Data
|
+----> Outbox Event
Then:
Outbox
|
v
Message Relay
|
v
Message Broker
This separates:
Transaction
from:
Message Delivery
while still guaranteeing that a committed business change has a corresponding event waiting to be published.
A Concrete Example
Suppose we have:
orders
and:
outbox
The tables might look like:
orders
+----+----------+----------+
| id | customer | status |
+----+----------+----------+
| 101| 5001 | CREATED |
+----+----------+----------+
The outbox might contain:
outbox
+----+----------------+------------------+
| id | event_type | payload |
+----+----------------+------------------+
| 900| OrderCreated | {...} |
+----+----------------+------------------+
The transaction is:
BEGIN;
INSERT INTO orders (
id,
customer_id,
status
)
VALUES (
101,
5001,
'CREATED'
);
INSERT INTO outbox (
id,
event_type,
payload
)
VALUES (
900,
'OrderCreated',
'{...}'
);
COMMIT;
Now both records are durable.
Only after the transaction commits does another component publish the event.
Why This Solves the Main Problem
Consider the original failure:
Database Commit
|
X
Application crashes
|
v
Event never published
With Outbox:
BEGIN
INSERT Order
INSERT OrderCreated into Outbox
COMMIT
|
X
Application crashes
The outbox record still exists.
When the publisher restarts:
Outbox
|
v
Publisher
|
v
Message Broker
The event can still be published.
The application crash no longer loses the event.
This is the fundamental advantage of the pattern.
The Outbox Table
A practical outbox table might look like:
CREATE TABLE outbox (
id UUID PRIMARY KEY,
aggregate_type VARCHAR(100) NOT NULL,
aggregate_id VARCHAR(100) NOT NULL,
event_type VARCHAR(200) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMP NOT NULL,
published_at TIMESTAMP NULL
);
For example:
+--------------------------------------+
| id |
+--------------------------------------+
| 7b8c... |
+--------------------------------------+
| aggregate_type = Order |
| aggregate_id = 12345 |
| event_type = OrderCreated |
| payload = {...} |
| created_at = ... |
| published_at = NULL |
+--------------------------------------+
The exact schema depends on the application.
But the important information is usually:
Event ID
Aggregate ID
Event Type
Payload
Creation Time
Publication State
Why Do We Need an Event ID?
Every event should have a unique identifier.
For example:
eventId = 9e6d2...
This becomes important because the Outbox Pattern does not necessarily provide exactly-once delivery.
A publisher can crash at the wrong moment.
Consider:
Read Outbox
|
v
Publish Event
|
v
Broker accepts Event
|
X
Publisher crashes
|
v
Mark event as published
The last step never happened.
When the publisher restarts:
Outbox says:
published = false
So it publishes the event again.
Now the broker receives:
OrderCreated
OrderCreated
The event was duplicated.
This is expected.
At-Least-Once Delivery
The practical delivery guarantee of an Outbox implementation is generally:
At least once
rather than:
Exactly once
The system tries to ensure:
If the database transaction commits, the event will eventually be published.
But the event may be published more than once.
Therefore:
Transactional Outbox
+
Idempotent Consumer
is an important combination.
Microservices.io explicitly calls out that the message relay can publish an event more than once if it crashes after publishing but before recording that it has done so, so consumers need to be idempotent.
The Publisher
The component responsible for moving events from the outbox to the broker is commonly called a:
Message Relay
or:
Outbox Publisher
The simplest implementation is polling.
Outbox
|
v
Poll Publisher
|
v
Message Broker
The publisher periodically executes:
SELECT *
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100;
Then:
for each event:
publish(event)
mark event as published
Conceptually:
while true:
events = readUnpublishedEvents()
for event in events:
publish(event)
markPublished(event)
This is called the Polling Publisher approach.
Polling Publisher
The architecture is:
Database
|
v
Outbox Table
|
SELECT unpublished
|
v
Outbox Publisher
|
v
Message Broker
The publisher might run every:
100 ms
or:
500 ms
or:
1 second
depending on latency requirements.
A shorter polling interval gives lower event latency but creates more database load.
A longer interval reduces database load but increases event delivery latency.
So there is a trade-off:
Polling frequency
|
+---- Lower latency
|
+---- Higher DB load
Batch Processing
A publisher should generally not process one event at a time.
Instead:
SELECT *
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100;
Then publish a batch:
Outbox
|
+-- Event 1
+-- Event 2
+-- Event 3
...
+-- Event 100
|
v
Publisher
|
v
Broker
Batching reduces:
Database queries
Network round trips
Broker calls
CPU overhead
But batch size should be controlled.
A huge batch can create:
Memory pressure
Long processing time
Large retry windows
Selecting Rows Safely
Multiple publisher instances may run simultaneously.
For example:
Publisher A
Publisher B
Publisher C
All are reading:
outbox
Without coordination, they might all select the same events.
For relational databases, row locking can help.
For example:
SELECT *
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED;
This allows multiple workers to process different rows concurrently.
Conceptually:
Outbox
|
+---- Event 1 ---- Publisher A
|
+---- Event 2 ---- Publisher B
|
+---- Event 3 ---- Publisher C
|
+---- Event 4 ---- Publisher A
The exact approach depends on the database and publishing architecture.
But Be Careful With Database Locks
You should not hold a database transaction open while waiting for the message broker.
Bad design:
BEGIN
SELECT event
FOR UPDATE
publish to Kafka
|
| wait
|
| network timeout
|
| retry
COMMIT
Now a database lock is held while an external system is contacted.
This can create:
Long transactions
Lock contention
Connection exhaustion
Poor throughput
The database should not be turned into a distributed transaction coordinator.
The Outbox Pattern exists partly to avoid that.
The Correct Mental Model for the Relay
Think of the outbox as a durable queue.
Application
|
| Local Transaction
v
Outbox
|
| Asynchronous Relay
v
Message Broker
The application says:
"This event must eventually be published."
The outbox stores that intention durably.
The relay handles:
"Publish this event to the broker."
This separation is powerful.
Polling vs Change Data Capture
Polling is not the only way to implement the relay.
Another approach is Change Data Capture (CDC).
Instead of continuously querying:
SELECT ...
a CDC system observes changes in the database transaction log or equivalent change stream.
Conceptually:
Application
|
v
Database
|
+---- Business Tables
|
+---- Outbox
|
v
Database Log
|
v
CDC
|
v
Message Broker
For relational databases, CDC can read changes from mechanisms such as transaction logs.
This avoids constantly polling the outbox table.
AWS describes both a traditional outbox-table approach and CDC-based implementations as ways to implement the pattern.
Polling vs CDC
The two approaches look like:
Polling:
Database
|
v
Outbox
|
v
Poller
|
v
Broker
and:
CDC:
Database
|
v
Transaction Log
|
v
CDC
|
v
Broker
Polling is:
Simple
Easy to understand
Easy to implement
Database-centric
CDC is:
More infrastructure
Lower polling overhead
Good for high-throughput systems
Often better suited to streaming architectures
Neither is universally better.
For a small system, polling may be more than sufficient.
For a large event-driven platform, CDC can become attractive.
The Most Important Failure Window
Consider:
Publisher
|
v
Read Event
|
v
Publish Event
|
X
CRASH
|
v
Mark Published
The broker already received the event.
But the outbox still says:
published_at = NULL
After restart:
Publisher
|
v
Read Event
|
v
Publish Again
So:
Event
↓
Broker
↓
Duplicate Event
This is why you should never build a consumer that assumes:
Exactly once
unless your entire messaging architecture genuinely provides and preserves that guarantee.
A safer design is:
At-least-once delivery
+
Idempotent consumers
Idempotent Consumers
Suppose the Payment Service receives:
OrderCreated
twice.
It should not create two payments.
A common approach is to store processed event IDs.
processed_events
+----------------------+
| event_id |
+----------------------+
| 9e6d... |
+----------------------+
The consumer performs:
BEGIN
if event_id already exists:
ignore event
else:
process event
INSERT event_id into processed_events
COMMIT
Now:
First delivery
|
v
Process event
|
v
Store event ID
Second delivery:
Second delivery
|
v
Event ID already exists
|
v
Ignore
This makes duplicate delivery harmless.
Outbox and Idempotency Solve Different Problems
These two patterns are often confused.
They solve different sides of the system.
Transactional Outbox
|
v
Reliable publication
while:
Idempotent Consumer
|
v
Safe duplicate processing
Together:
Producer
|
v
Outbox
|
v
Broker
|
v
Idempotent Consumer
The Outbox protects against:
Database committed
but event lost
Idempotency protects against:
Event delivered more than once
You often need both.
Ordering
Ordering is another important concern.
Suppose an Order goes through:
OrderCreated
OrderPaid
OrderShipped
The consumer should not receive:
OrderShipped
OrderCreated
OrderPaid
The correct logical order is:
OrderCreated
↓
OrderPaid
↓
OrderShipped
The Outbox gives you a place to establish publication order.
For example:
+-----+----------------+
| seq | event |
+-----+----------------+
| 101 | OrderCreated |
| 102 | OrderPaid |
| 103 | OrderShipped |
+-----+----------------+
The relay publishes them in order.
Microservices.io notes that preserving the order in which events were generated is an important requirement, particularly when multiple transactions update the same aggregate.
But there is an important distinction:
Producer-side ordering does not automatically guarantee consumer-side ordering.
The message broker and consumer architecture must also preserve the required ordering.
Ordering Per Aggregate
Global ordering is usually unnecessary.
Imagine:
Order 100
Order 101
Order 102
There is generally no reason that:
OrderCreated(100)
must be processed before:
OrderCreated(101)
What usually matters is ordering within the same aggregate.
For example:
Order 100
OrderCreated
↓
OrderPaid
↓
OrderShipped
while independently:
Order 101
OrderCreated
↓
OrderPaid
Both workflows can proceed independently.
This allows much better scalability.
Partitioning
With Kafka-like systems, one common strategy is to partition events by aggregate ID.
For example:
partition = hash(orderId)
Then:
Order 100 → Partition 1
Order 101 → Partition 2
Order 102 → Partition 1
Order 103 → Partition 3
Events for the same order go to the same partition:
Order 100
OrderCreated
OrderPaid
OrderShipped
This allows the broker to preserve ordering for that aggregate while still processing many aggregates concurrently.
The exact implementation depends on the messaging platform.
What Should the Outbox Store?
There are two broad choices.
Store the Complete Event
outbox
event_type = OrderCreated
payload = {
"orderId": 123,
"customerId": 456,
"items": [...]
}
Advantages:
Simple publishing
No need to reconstruct the event later
Stable event payload
Disadvantages:
More storage
Potentially duplicated data
Store Event Metadata
Alternatively:
outbox
event_type = OrderCreated
aggregate_id = 123
The publisher could retrieve the actual data later.
But this introduces another problem.
The data may have changed since the event was created.
For example:
OrderCreated
was generated when:
status = CREATED
Later:
status = CANCELLED
If the publisher reconstructs the event from the current database state, it may accidentally publish:
OrderCreated
status = CANCELLED
That is incorrect.
For this reason, storing the event payload at the time of the transaction is often safer.
Event Versioning
Once events are published, consumers may depend on their structure.
Suppose version 1 is:
{
"orderId": 123,
"customerId": 456
}
Later you add:
{
"orderId": 123,
"customerId": 456,
"currency": "USD"
}
Existing consumers must continue to work.
Therefore, event schemas should be treated as contracts.
Common approaches include:
Backward-compatible changes
Schema versioning
Schema registry
Explicit event versions
For example:
eventType = OrderCreated
eventVersion = 2
The Outbox is not responsible for schema evolution, but because it becomes the durable source of outgoing events, event schema design becomes part of the overall architecture.
Outbox Payload Size
Be careful about putting huge payloads into the outbox.
For example:
OrderCreated
|
+-- 10 MB product metadata
+-- images
+-- documents
+-- customer profile
This creates:
Large database rows
Large transaction sizes
More storage
Slower replication
Slower CDC
More broker traffic
A good event usually contains the information consumers need to understand the event.
For example:
{
"eventId": "9e6d...",
"eventType": "OrderCreated",
"orderId": "12345",
"occurredAt": "2026-09-02T10:30:00Z"
}
Consumers can retrieve additional information through appropriate APIs or projections when necessary.
Outbox Cleanup
The outbox grows continuously.
Suppose the application generates:
10,000 events / minute
Then:
10,000
×
60
×
24
creates a large number of rows every day.
So published events need a retention strategy.
For example:
Outbox
|
+-- Unpublished
|
+-- Published < 1 hour
|
+-- Published < 1 day
|
+-- Archived
|
+-- Deleted
One approach is:
DELETE FROM outbox
WHERE published_at < now() - interval '7 days';
Another is partitioning:
outbox_2026_09_01
outbox_2026_09_02
outbox_2026_09_03
Then old partitions can be dropped efficiently.
The right strategy depends on:
Retention requirements
Audit requirements
Replay requirements
Database size
Event volume
Compliance
Do Not Delete Too Early
Suppose:
Event published
and immediately:
DELETE event
If you later discover a consumer problem, you may have lost the ability to inspect or replay the event.
Therefore, you need to decide whether the outbox is:
Temporary delivery queue
or:
Historical event archive
Usually, it should not be treated as a permanent event store.
If you need a durable event history, use an appropriate event-streaming or event-sourcing architecture.
The outbox's primary job is:
Reliable handoff
not:
Permanent event storage
Monitoring the Outbox
The outbox introduces a new operational component.
You now need to monitor:
Outbox size
Oldest unpublished event
Publishing latency
Publishing failures
Retry count
Dead-letter events
Publisher throughput
One particularly useful metric is:
Outbox Lag
For example:
Current time:
10:30:00
Oldest unpublished event:
10:29:52
Outbox lag:
8 seconds
If this suddenly becomes:
10 minutes
something is wrong.
Possible causes:
Publisher is down
Database is slow
Broker is unavailable
Network failure
Publisher backlog
Poison event
Outbox lag is therefore a very useful health indicator.
The Outbox Is a Queue
It is helpful to think of the outbox as a durable queue inside your database.
Application
|
v
+-------------+
| Database |
| |
| Business DB |
| |
| Outbox |
+-------------+
|
v
Publisher
|
v
Message Broker
The database transaction guarantees:
Business State
+
Event Intent
are committed together.
The publisher guarantees:
Event Intent
|
v
Message Broker
happens asynchronously.
This separation is the core architectural idea.
Outbox With Saga
This is where the previous pattern in this series becomes important.
A Saga often looks like:
Order Service
|
| OrderCreated
v
Payment Service
|
| PaymentCompleted
v
Inventory Service
How does Order Service reliably publish:
OrderCreated
after creating the order?
Use Outbox.
Order Service
BEGIN
Create Order
Insert OrderCreated into Outbox
COMMIT
Then:
Outbox
|
v
Publisher
|
v
Message Broker
|
v
Payment Service
The Payment Service does the same thing:
BEGIN
Update Payment
Insert PaymentCompleted into Outbox
COMMIT
Then:
Payment Outbox
|
v
Message Broker
|
v
Inventory Service
So the architecture becomes:
Saga
|
+---------------+---------------+
| | |
v v v
Order Service Payment Service Inventory Service
| | |
DB DB DB
| | |
Outbox Outbox Outbox
| | |
+---------------+---------------+
|
v
Message Broker
This is one of the most common combinations of distributed-system patterns.
The Saga defines:
What should happen next?
The Outbox solves:
How do I reliably publish that state transition?
Saga Without Outbox
Consider:
Order Service
|
+---- Create Order
|
+---- Publish OrderCreated
Without Outbox:
Create Order ✓
|
X
Application crashes
|
v
OrderCreated never published
The Saga is now stuck.
The Order Service believes:
Order = CREATED
but the Payment Service never receives:
OrderCreated
So the Saga cannot progress.
Saga With Outbox
Now:
BEGIN
Create Order
Write OrderCreated to Outbox
COMMIT
Then:
Application crashes
No problem.
The outbox still contains:
OrderCreated
When the publisher restarts:
Outbox
|
v
OrderCreated
|
v
Message Broker
|
v
Payment Service
The Saga continues.
This is why the Outbox Pattern is such an important companion to Saga.
Outbox vs Event Sourcing
These patterns are sometimes confused.
They are fundamentally different.
Transactional Outbox
The primary source of truth is still:
Current State
For example:
orders
The outbox is used to reliably publish events:
orders
+
outbox
Event Sourcing
The events themselves are the source of truth.
For example:
OrderCreated
OrderPaid
OrderShipped
OrderCancelled
The current state can be reconstructed from the event history.
Conceptually:
Events
|
+-- OrderCreated
+-- OrderPaid
+-- OrderShipped
|
v
Current State
So:
Outbox
=
Current state + reliable event publication
while:
Event Sourcing
=
Events as the source of truth
They solve different problems.
Outbox vs Database Triggers
Another approach is using database triggers.
For example:
INSERT INTO orders
|
v
Database Trigger
|
v
INSERT INTO outbox
This can guarantee that changes result in outbox entries.
But triggers introduce coupling between:
Database
+
Business behavior
They can also make application behavior harder to understand and test.
The application-level approach is often easier to reason about:
Service
|
+-- Update entity
|
+-- Create event
|
+-- Commit
The right choice depends on architecture and operational requirements.
Common Outbox Mistakes
Mistake 1: Publishing Directly After the Database Commit
save()
commit()
publish()
This still has the crash window:
commit
|
X
crash
|
publish never happens
Use an outbox.
Mistake 2: Publishing Before the Database Transaction
publish()
save()
commit()
Now the event can exist even when the database transaction fails.
Again, use an outbox.
Mistake 3: Assuming Exactly Once
The publisher can crash after publishing.
Therefore:
Outbox
+
Publisher
can produce duplicate messages.
Consumers must be idempotent.
Mistake 4: Holding Database Locks While Publishing
Avoid:
BEGIN
SELECT FOR UPDATE
publish to broker
COMMIT
The external network call should not hold database locks.
Mistake 5: Ignoring Ordering
If:
OrderCreated
OrderPaid
OrderCancelled
are published out of order, consumers can reach invalid states.
Ordering requirements should be explicitly designed.
Mistake 6: Never Cleaning the Outbox
An outbox that is never cleaned eventually becomes a database-storage problem.
Treat:
Retention
Partitioning
Archival
Cleanup
as part of the design.
Mistake 7: Putting Everything Into the Event
Do not turn every event into a massive snapshot of your entire database.
Events should have purposeful contracts.
A Production-Ready Flow
A robust implementation looks like:
Client
|
v
Order Service
|
v
BEGIN TRANSACTION
|
+---------+---------+
| |
v v
Orders Table Outbox Table
| |
+---------+---------+
|
COMMIT
|
v
Outbox Publisher
|
+-----+-----+
| |
Success Failure
| |
v v
Broker Retry
|
v
Idempotent Consumer
|
v
Downstream Service
The important properties are:
Business update
+
Outbox insert
are atomic.
Then:
Outbox
|
v
Asynchronous publication
is retried independently.
And:
Consumer
is idempotent.
Failure Scenarios
Let's walk through the important failures.
Database Transaction Fails
Create Order
|
v
Insert Outbox
|
X
Transaction Rollback
Result:
Order does not exist
Outbox event does not exist
Correct.
Nothing should be published.
Application Crashes Before Commit
Create Order
Insert Outbox
|
X
Crash
Result:
Transaction rolled back
No event should be published.
Correct.
Application Crashes After Commit
Create Order
Insert Outbox
|
v
COMMIT
|
X
Crash
Result:
Order exists
Outbox event exists
Publisher eventually sends the event.
Correct.
Broker Is Down
Outbox
|
v
Publisher
|
X
Broker unavailable
The event remains in the outbox.
The publisher retries later.
Correct.
Publisher Crashes After Publish
Outbox
|
v
Publisher
|
v
Broker ✓
|
X
Publisher crashes
The event may be published again.
Therefore:
Consumer must be idempotent
Correct.
Consumer Crashes
Broker
|
v
Consumer
|
v
Process Event
|
X
Crash before acknowledgement
The broker may deliver the event again.
Again:
Idempotent Consumer
is required.
The Complete Reliability Chain
This leads to a very useful mental model:
Local Transaction
|
+---------+---------+
| |
v v
Business Data Outbox
|
v
Message Relay
|
v
Broker
|
v
Consumer
|
+------+------+
| |
v v
Process Event Duplicate?
| |
v v
Commit Ignore
Each component handles a different failure.
Database Transaction
↓
Atomic local state change
Outbox
↓
Durable event intent
Relay
↓
Reliable publication
Broker
↓
Message transport
Idempotent Consumer
↓
Safe duplicate processing
This is how a collection of relatively simple patterns creates a reliable distributed workflow.
Exactly-Once Is Usually the Wrong Goal
One of the biggest traps in distributed systems is obsessing over:
Exactly once
Instead, design for:
At least once
+
Idempotency
Why?
Because distributed failures can occur between any two operations.
For example:
Publish
|
v
Broker
|
v
ACK
A network failure can happen between:
Broker accepted message
and:
Publisher received ACK
The publisher cannot know whether the broker accepted the message.
So it retries.
Now you have:
Maybe once
Maybe twice
The practical solution is not to pretend this uncertainty does not exist.
The practical solution is:
Duplicate-safe processing.
Outbox and Exactly-Once Business Effects
There is an important distinction between:
Exactly-once message delivery
and:
Exactly-once business effect
You may receive:
OrderCreated
OrderCreated
OrderCreated
three times.
But if the consumer is idempotent:
First message
↓
Create Order Projection
Second message
↓
Already processed
↓
Ignore
Third message
↓
Already processed
↓
Ignore
the business effect happens only once.
That is usually what you actually care about.
Testing the Outbox
The Outbox Pattern introduces failure scenarios that should be explicitly tested.
For example:
Test 1:
Database transaction fails
Expected:
No business data
No outbox event
Test 2:
Application crashes after commit
Expected:
Business data exists
Outbox event exists
Test 3:
Broker unavailable
Expected:
Event remains unpublished
Publisher retries
Test 4:
Publisher crashes after publishing
Expected:
Duplicate event possible
Consumer handles duplicate safely
Test 5:
Consumer crashes before acknowledgment
Expected:
Event redelivered
Consumer remains correct
These are not unusual edge cases.
They are normal distributed-system behavior.
A Simple Java Implementation
A simplified Spring-style implementation might look like:
@Transactional
public void createOrder(CreateOrderCommand command) {
Order order = new Order(
command.customerId(),
command.items()
);
orderRepository.save(order);
OutboxEvent event = OutboxEvent.of(
UUID.randomUUID(),
"OrderCreated",
order.getId(),
createPayload(order)
);
outboxRepository.save(event);
}
Notice what is missing.
There is no:
kafkaTemplate.send(...)
inside this transaction.
Instead, the application only writes:
Order
+
OutboxEvent
to the same database transaction.
Then a separate publisher handles:
@Scheduled(...)
public void publishEvents() {
List<OutboxEvent> events =
outboxRepository.findUnpublished();
for (OutboxEvent event : events) {
kafkaTemplate.send(
"orders",
event.getAggregateId(),
event.getPayload()
);
outboxRepository.markPublished(
event.getId()
);
}
}
This is simplified code.
A production implementation needs to deal with:
Concurrent publishers
Retries
Duplicate publication
Ordering
Broker failures
Batching
Locking
Cleanup
Metrics
Tracing
Dead-letter handling
But the fundamental idea remains the same.
The Most Important Design Rule
When a service needs to perform:
Update database
+
Publish event
do not think of them as two independent operations.
Think of them as:
One business transaction
and put both pieces of intent inside the same local transaction:
BEGIN
Update Business State
Create Outbox Event
COMMIT
Then publish asynchronously.
This gives you:
Atomic local state change
+
Durable event intent
without requiring a distributed transaction.
When Should You Use the Outbox Pattern?
The pattern is especially useful when:
Service updates its database
+
Other services need to know about that change
Examples include:
OrderCreated
PaymentCompleted
InventoryReserved
UserRegistered
SubscriptionCancelled
InvoiceGenerated
ShipmentCreated
It is particularly useful in:
Microservices
Event-driven architectures
Saga workflows
Domain event publishing
Asynchronous integrations
The pattern is designed specifically for the situation where a service must atomically update its local data and send a message/event.
When You May Not Need It
Not every database operation needs an outbox.
If:
Update database
does not require:
Publish event
then there is no dual-write problem.
Similarly, if a system already has a storage mechanism where the state change itself naturally produces a reliable change stream, another architecture may be more appropriate.
The important question is:
Do I need to reliably communicate a committed state change to another system?
If yes, Outbox is one of the patterns worth considering.
Outbox Is Not a Message Broker
Another important distinction:
Outbox
is not intended to replace:
Kafka
RabbitMQ
SQS
Pulsar
The outbox is a durable bridge between the database transaction and the messaging system.
Think of it as:
Database
|
| Reliable handoff
v
Outbox
|
| Asynchronous publication
v
Message Broker
The broker still handles:
Distribution
Fan-out
Consumer management
Delivery
Partitioning
Backpressure
Retention
Replay
The outbox handles:
Atomicity between local state and outgoing event
A Useful Mental Model
Imagine a restaurant.
The database is the kitchen's order system.
The message broker is the waiter.
You do not want:
Kitchen records order
|
X
Waiter never receives it
Instead, the kitchen first writes:
Order
+
"Tell waiter about this order"
into its own durable system.
Then the waiter picks up the notification.
If the waiter temporarily disappears:
Order
+
Notification
still exists.
The notification can be delivered later.
That is essentially what the Outbox Pattern does.
Saga + Outbox + Idempotency
These three patterns form a particularly useful combination.
Saga
|
v
Business Workflow
|
+----------+----------+
| |
v v
Local Transaction Compensation
|
v
Outbox
|
v
Broker
|
v
Idempotent Consumer
Each solves a different problem:
Saga
|
+-- Coordinates distributed business transactions
Outbox
|
+-- Reliably publishes local state changes
Idempotency
|
+-- Makes duplicate delivery safe
Together:
Saga
+
Transactional Outbox
+
Idempotent Consumers
form a powerful foundation for event-driven microservices.
Final Takeaway
The Transactional Outbox Pattern solves a deceptively simple problem:
How do I update my database and reliably publish an event without using a distributed transaction?
The naïve approach:
Database
+
Message Broker
creates a dual-write problem.
The Outbox approach changes the architecture:
BEGIN
Update Business Data
Write Event to Outbox
COMMIT
Then:
Outbox
|
v
Message Relay
|
v
Message Broker
This gives us:
Database update
+
Event intent
as one atomic local transaction.
But it does not give us exactly-once delivery.
The relay can publish duplicates.
Therefore:
Transactional Outbox
+
At-Least-Once Delivery
+
Idempotent Consumers
is the practical model.
The complete picture is:
Command
|
v
Service Transaction
|
+--------+--------+
| |
v v
Business Data Outbox Event
| |
+--------+--------+
|
COMMIT
|
v
Message Relay
|
v
Message Broker
|
+--------+--------+
| |
v v
Consumer A Consumer B
| |
v v
Idempotent Idempotent
Processing Processing
And the deepest lesson is:
Do not try to make your database and message broker behave like one transactional system. Make the database transaction authoritative, persist the intent to publish alongside the business state, and let a reliable asynchronous process deliver that intent to the broker.
Once you understand this pattern, the relationship between several distributed-system patterns becomes much clearer:
Saga
|
| needs reliable events
v
Transactional Outbox
|
| produces at-least-once messages
v
Idempotent Consumer
|
| handles duplicates safely
v
Reliable Distributed Workflow
That is the real value of the Outbox Pattern.
It does not eliminate distributed-system failures.
It moves the failure boundary into a place where the system can recover from it safely.