The Problem: One Business Operation, Multiple Services
Imagine an e-commerce application.
A customer places an order:
Place Order
|
+----> Reserve Inventory
|
+----> Process Payment
|
+----> Create Shipment
|
+----> Confirm Order
In a monolithic application, this might be relatively straightforward.
You could have a single database transaction:
BEGIN TRANSACTION
Create Order
Reserve Inventory
Process Payment
Create Shipment
COMMIT
If something fails:
BEGIN TRANSACTION
Create Order
Reserve Inventory
Process Payment
Payment fails
ROLLBACK
The database takes care of atomicity.
Everything is committed together.
Or nothing is committed.
The problem becomes much harder when the application is split into microservices.
For example:
Order Service
|
Order DB
|
|
---------------------
| |
v v
Inventory Service Payment Service
| |
Inventory DB Payment DB
|
v
Shipping Service
|
Shipping DB
Now there is no single database transaction covering the entire operation.
The order database belongs to the Order Service.
The inventory database belongs to the Inventory Service.
The payment database belongs to the Payment Service.
The shipping database belongs to the Shipping Service.
Each service controls its own local transaction.
So the original problem becomes:
How do we maintain consistency when one business operation spans multiple independent services and databases?
This is the problem the Saga Pattern addresses.
What Is the Saga Pattern?
A Saga is a sequence of local transactions.
Each service performs its own local transaction.
If all steps succeed, the business operation succeeds.
If a later step fails, previously completed steps are compensated using compensating transactions.
Conceptually:
T1 → T2 → T3 → T4
|
X
|
↓
C3 → C2 → C1
Where:
T1 = Local Transaction 1
T2 = Local Transaction 2
T3 = Local Transaction 3
T4 = Local Transaction 4
C1 = Compensation for T1
C2 = Compensation for T2
C3 = Compensation for T3
The important idea is:
A Saga does not provide one global ACID transaction.
Instead, it turns one large distributed transaction into multiple local transactions and explicitly defines how the system should recover when something goes wrong.
This distinction is fundamental.
A Saga is not a distributed database transaction.
It is a distributed business workflow with explicit compensation.
Why Do We Need Saga?
Consider an order workflow:
Create Order
↓
Reserve Inventory
↓
Charge Payment
↓
Create Shipment
Suppose each operation takes place in a different service.
The workflow might look like:
Order Service
|
| Create Order
↓
Inventory Service
|
| Reserve Inventory
↓
Payment Service
|
| Charge Customer
↓
Shipping Service
|
| Create Shipment
↓
Success
Now imagine this happens:
Create Order ✓
Reserve Inventory ✓
Charge Payment ✓
Create Shipment ✗
We now have a problem.
The customer has already been charged.
Inventory has already been reserved.
The order exists.
But the shipment could not be created.
We cannot simply execute:
ROLLBACK
because there is no single transaction.
The payment database has already committed.
The inventory database has already committed.
The order database has already committed.
The shipping database failed.
This is where Saga enters the picture.
We define compensating operations:
Create Order
↓
Reserve Inventory
↓
Charge Payment
↓
Create Shipment ✗
↓
Refund Payment
↓
Release Inventory
↓
Cancel Order
The system does not magically roll back.
The application explicitly performs actions that compensate for the previously completed business operations.
Saga Is Not Rollback
This is one of the most important concepts to understand.
A database rollback means:
Transaction
|
+-- UPDATE
+-- INSERT
+-- DELETE
|
X
|
ROLLBACK
The database restores the previous transactional state.
A Saga does something fundamentally different.
Suppose:
Charge Payment
succeeds.
Later:
Create Shipment
fails.
The Saga cannot rollback the payment transaction.
Instead, it performs another business operation:
Refund Payment
So:
Charge Payment
↓
Payment committed
↓
Shipment fails
↓
Refund Payment
The refund is a new transaction.
Therefore:
Database rollback
≠
Saga compensation
A compensating transaction is a business operation that semantically reverses or compensates for an earlier operation.
This distinction becomes extremely important when designing real systems.
A Complete Example
Let's build a simple order-processing Saga.
Suppose we have four services:
Order Service
Inventory Service
Payment Service
Shipping Service
Each service owns its own database.
+----------------+
| Order Service |
+----------------+
|
Order DB
+---------------------+
| Inventory Service |
+---------------------+
|
Inventory DB
+------------------+
| Payment Service |
+------------------+
|
Payment DB
+--------------------+
| Shipping Service |
+--------------------+
|
Shipping DB
The business workflow is:
1. Create Order
2. Reserve Inventory
3. Charge Payment
4. Create Shipment
5. Confirm Order
We can represent it as:
T1: Create Order
↓
T2: Reserve Inventory
↓
T3: Charge Payment
↓
T4: Create Shipment
↓
T5: Confirm Order
Now define compensations:
T1: Create Order
C1: Cancel Order
T2: Reserve Inventory
C2: Release Inventory
T3: Charge Payment
C3: Refund Payment
T4: Create Shipment
C4: Cancel Shipment
If T4 fails:
T1 ✓
↓
T2 ✓
↓
T3 ✓
↓
T4 ✗
The Saga performs:
C3
↓
C2
↓
C1
So:
Create Order
↓
Reserve Inventory
↓
Charge Payment
↓
Create Shipment ✗
↓
Refund Payment
↓
Release Inventory
↓
Cancel Order
This is the basic Saga model.
Local Transactions
The word local is extremely important.
Each step is a normal transaction inside one service.
For example:
Inventory Service
BEGIN
UPDATE inventory
SET available = available - 1
WHERE product_id = 100
AND available > 0;
COMMIT
The Inventory Service does not participate in the Payment Service's database transaction.
The Payment Service independently executes:
BEGIN
INSERT INTO payments (...);
COMMIT
The Order Service independently executes:
BEGIN
INSERT INTO orders (...);
COMMIT
The Saga connects these independent transactions into one business workflow.
Conceptually:
Saga
|
+-------+-------+
| | |
v v v
T1 T2 T3
| | |
DB1 DB2 DB3
Each transaction is locally atomic.
The Saga provides coordination across them.
The Two Ways to Implement a Saga
There are two major approaches:
Saga
|
+-- Choreography
|
+-- Orchestration
The difference is primarily about who controls the workflow.
Choreography-Based Saga
In choreography, there is no central coordinator.
Services communicate through events.
Each service:
-
Performs its local transaction.
-
Publishes an event.
-
Another service reacts to that event.
-
That service performs its own local transaction.
-
It publishes another event.
The workflow emerges from the interaction between services.
For example:
Order Service
|
| OrderCreated
↓
Payment Service
|
| PaymentCompleted
↓
Inventory Service
|
| InventoryReserved
↓
Shipping Service
|
| ShipmentCreated
There is no:
Saga Coordinator
Instead, the services react to events.
Choreography Example
Suppose the Order Service creates an order.
It commits:
Order Status = PENDING
Then publishes:
OrderCreated
The Payment Service receives:
OrderCreated
It processes the payment.
If successful:
Payment Status = COMPLETED
Then publishes:
PaymentCompleted
The Inventory Service receives:
PaymentCompleted
It reserves inventory.
If successful:
Inventory Status = RESERVED
Then publishes:
InventoryReserved
The Shipping Service receives:
InventoryReserved
and creates the shipment.
The complete flow becomes:
Order Service
|
| OrderCreated
↓
Payment Service
|
| PaymentCompleted
↓
Inventory Service
|
| InventoryReserved
↓
Shipping Service
|
| ShipmentCreated
The message broker might look like:
Message Broker
|
+--------------+--------------+
| | |
v v v
Order Service Payment Service Inventory Service
| | |
+--------------+--------------+
|
Shipping Service
The services are loosely coupled through events.
Choreography Failure Handling
Now suppose inventory reservation fails.
The flow becomes:
OrderCreated
↓
PaymentCompleted
↓
InventoryReservationFailed
The Payment Service can listen for:
InventoryReservationFailed
and execute:
Refund Payment
Then publish:
PaymentRefunded
The Order Service can react:
PaymentRefunded
↓
Cancel Order
So compensation also happens through events.
Conceptually:
OrderCreated
↓
PaymentCompleted
↓
InventoryReservationFailed
↓
PaymentRefunded
↓
OrderCancelled
This is elegant for small workflows.
But there is a price.
The Problem With Choreography
Consider a Saga with only three services:
A → B → C
It is relatively easy to understand.
Now imagine:
A → B → C → D → E → F → G
And every service reacts to several events.
Soon the architecture starts looking like:
A ─────→ B ─────→ C
│ │ │
│ ↓ ↓
│ D ─────→ E
│ │ │
↓ ↓ ↓
F ─────→ G ─────→ H
The workflow is no longer obvious.
To understand what happens when:
InventoryReservationFailed
you may have to inspect multiple services.
You need to know:
Who publishes the event?
Who consumes it?
What transaction do they execute?
What event do they publish?
Who consumes that event?
What happens if that service fails?
What compensates the previous transaction?
The business workflow becomes distributed across the codebase.
This is the primary weakness of choreography.
As the Saga grows, the system can develop a distributed workflow that is difficult to visualize and debug. This is one reason orchestration becomes attractive for more complex workflows.
Orchestration-Based Saga
Orchestration introduces a central component:
Saga Orchestrator
The orchestrator knows the workflow.
Instead of services discovering what to do by reacting to events, the orchestrator sends commands.
The architecture becomes:
Saga Orchestrator
|
+-------------+-------------+
| | |
v v v
Order Service Payment Service Inventory Service
|
v
Shipping Service
The orchestrator might execute:
Create Order
↓
Reserve Inventory
↓
Charge Payment
↓
Create Shipment
↓
Confirm Order
The orchestrator knows the entire process.
Orchestration Example
The Saga starts:
POST /orders
The Order Service creates an order.
Then the orchestrator sends:
ReserveInventory
to the Inventory Service.
Inventory responds:
InventoryReserved
The orchestrator sends:
ChargePayment
to the Payment Service.
Payment responds:
PaymentCompleted
The orchestrator sends:
CreateShipment
to the Shipping Service.
Shipping responds:
ShipmentCreated
Finally:
ConfirmOrder
The workflow is:
Orchestrator
|
|
ReserveInventory
|
v
Inventory
|
|
InventoryReserved
|
v
Orchestrator
|
ChargePayment
|
v
Payment
|
|
PaymentCompleted
|
v
Orchestrator
|
CreateShipment
|
v
Shipping
|
|
ShipmentCreated
The workflow is much easier to see.
Orchestration Failure Handling
Suppose:
Create Order ✓
Reserve Inventory ✓
Charge Payment ✓
Create Shipment ✗
The orchestrator knows exactly what has already happened.
It can execute:
Refund Payment
↓
Release Inventory
↓
Cancel Order
The complete state machine becomes:
START
|
v
Create Order
|
v
Reserve Inventory
|
v
Charge Payment
|
v
Create Shipment
/ \
✓ X
| |
v v
SUCCESS Refund Payment
|
v
Release Inventory
|
v
Cancel Order
|
v
FAILED
This is one of the major advantages of orchestration.
The business workflow exists in one place.
Choreography vs Orchestration
The difference can be summarized as:
Choreography
Service A
|
| Event
↓
Service B
|
| Event
↓
Service C
versus:
Orchestration
Orchestrator
/ | \
↓ ↓ ↓
A B C
Choreography says:
"Something happened. Whoever cares about it should react."
Orchestration says:
"Here is the next operation that needs to happen."
Neither approach is universally better.
The choice depends on workflow complexity, coupling, operational requirements, and how much centralized visibility you want.
When Choreography Works Well
Choreography can be a good choice when:
Small number of services
+
Simple workflow
+
Loose coupling
+
Event-driven architecture
For example:
OrderCreated
↓
Payment
↓
PaymentCompleted
↓
Notification
The workflow is simple enough that the event relationships remain understandable.
Advantages include:
No central coordinator
↓
No central workflow component
↓
Services remain autonomous
↓
Natural event-driven architecture
There is also no single orchestrator that becomes a central control point.
When Choreography Becomes Difficult
Choreography becomes increasingly difficult when:
Many services
+
Many events
+
Many compensation paths
+
Complex business rules
+
Long-running workflows
For example:
OrderCreated
↓
PaymentCompleted
↓
InventoryReserved
↓
FraudCheckPassed
↓
CreditApproved
↓
WarehouseAllocated
↓
ShipmentCreated
↓
NotificationSent
Now imagine each step can fail.
The number of possible paths increases rapidly.
You might have:
PaymentFailed
InventoryFailed
FraudFailed
CreditFailed
WarehouseFailed
ShipmentFailed
NotificationFailed
and each failure can trigger compensation.
At some point, the workflow becomes difficult to understand by looking at events alone.
When Orchestration Works Well
Orchestration is particularly useful when the business workflow is complex.
For example:
Order
|
+-- Validate
|
+-- Reserve Inventory
|
+-- Authorize Payment
|
+-- Fraud Check
|
+-- Reserve Warehouse
|
+-- Create Shipment
|
+-- Confirm Order
The orchestrator can explicitly model this state machine.
Order Saga
|
+----------------+----------------+
| | |
Validate Payment Inventory
| | |
+----------------+----------------+
|
Fraud Check
|
Warehouse
|
Shipment
|
Confirm
The workflow is easier to reason about.
The Most Important Part: Compensation
Many explanations of Saga focus on:
Transaction 1
Transaction 2
Transaction 3
But the difficult part is actually:
What happens when Transaction 3 fails?
This is where Saga design becomes difficult.
For every forward operation, you should ask:
What happens if a later operation fails?
For example:
Reserve Inventory
might have:
Compensation:
Release Inventory
Payment:
Charge Payment
might have:
Compensation:
Refund Payment
Order:
Create Order
might have:
Compensation:
Cancel Order
So the Saga definition should contain both:
Forward Action
+
Compensating Action
A useful design table is:
+----------------------+----------------------+
| Forward Transaction | Compensation |
+----------------------+----------------------+
| Create Order | Cancel Order |
| Reserve Inventory | Release Inventory |
| Charge Payment | Refund Payment |
| Create Shipment | Cancel Shipment |
+----------------------+----------------------+
If you cannot define a meaningful compensation, that operation deserves special attention.
Compensation Is Not Always Perfectly Reversible
Suppose a payment was charged.
A refund is not necessarily identical to a rollback.
For example:
Charge $100
followed later by:
Refund $100
The system has not returned to exactly the same state.
There may have been:
Payment processing fees
Ledger entries
Notifications
Fraud events
Audit records
External side effects
Therefore, compensation should be thought of as:
Restore the business invariant.
Not:
Restore the exact previous database state.
This is a crucial distinction.
Irreversible Operations
Some operations are difficult or impossible to compensate.
For example:
Send Email
You cannot really execute:
Undo Email
Similarly:
Send SMS
cannot be perfectly reversed.
Or:
Notify External Partner
may already have triggered another workflow.
Therefore, not every step in a Saga should necessarily be treated as a normal reversible transaction.
You need to understand the business semantics of every side effect.
Saga and Eventual Consistency
A Saga usually means accepting eventual consistency across services.
Suppose:
Order Service
has:
status = PENDING
while the Saga is running.
Then:
Payment Service
might already have:
status = COMPLETED
while:
Inventory Service
is still processing.
For a short period:
Order = PENDING
Payment = COMPLETED
Inventory = UNKNOWN
The system is temporarily inconsistent from a global perspective.
Eventually:
Order = CONFIRMED
Payment = COMPLETED
Inventory = RESERVED
Shipment = CREATED
or:
Order = CANCELLED
Payment = REFUNDED
Inventory = RELEASED
Shipment = NONE
This is an important mental model:
A Saga does not make the entire distributed system instantly consistent.
It drives the system toward a valid business state.
The Missing ACID Property: Isolation
A Saga gives you local atomicity, but it does not automatically provide global isolation.
Consider:
Saga A:
Reserve Inventory
and simultaneously:
Saga B:
Reserve Inventory
Both workflows might observe state that is valid locally but problematic globally.
For example:
Inventory = 1 item
Saga A → sees 1 available
Saga B → sees 1 available
If concurrency is not handled correctly, both might attempt to reserve it.
Therefore, Saga design often needs additional techniques such as:
Optimistic concurrency
Pessimistic locking
Version numbers
State machines
Reservation records
Semantic locks
Idempotency
The Saga pattern itself does not magically solve concurrency.
This is one of the major differences between a Saga and a traditional database transaction. The lack of global isolation is a known Saga trade-off.
Semantic Locking
One useful technique is to represent an in-progress operation explicitly.
For example:
Inventory:
AVAILABLE
|
v
RESERVED
|
v
CONFIRMED
Instead of immediately treating inventory as permanently sold, the system creates a reservation.
For example:
Inventory Reservation
product_id = 100
order_id = 5001
status = RESERVED
expires_at = ...
Now other operations understand that the inventory is temporarily reserved.
This is often much easier to reason about than pretending the entire distributed workflow is one atomic transaction.
Idempotency Is Critical
Distributed systems retry.
Messages can be delivered more than once.
Requests can be retried.
Consumers can crash after processing a message but before acknowledging it.
Therefore, Saga operations should usually be idempotent.
Consider:
ReserveInventory(orderId=5001)
The message is delivered twice:
Message 1 → Reserve Inventory
Message 2 → Reserve Inventory
If the operation is not idempotent, inventory might be reserved twice.
Instead, the service should recognize:
orderId = 5001
has already been processed.
For example:
if reservation already exists:
return existing reservation
else:
create reservation
The desired property is:
Execute once
or:
Execute twice
or:
Execute ten times
all produce the same business result.
This is extremely important in Saga implementations.
The Outbox Problem
There is another subtle failure scenario.
Suppose the Order Service does:
BEGIN TRANSACTION
INSERT INTO orders (...)
COMMIT
and then publishes:
OrderCreated
What happens if the service crashes between those operations?
Database Commit
↓
CRASH
↓
Publish Event
The order exists.
But the event was never published.
Now the Saga is stuck.
The reverse problem can also happen:
Publish Event
↓
CRASH
↓
Database Commit
The event may exist even though the database transaction did not commit.
This is the classic dual-write problem.
A common solution is the Transactional Outbox Pattern.
Saga + Transactional Outbox
Instead of:
Database
+
Message Broker
being two independent operations, the service writes the business data and an outgoing event into the same local database transaction.
For example:
BEGIN TRANSACTION
INSERT INTO orders (...)
INSERT INTO outbox (
event_type,
payload
)
COMMIT
Now both are committed atomically.
A separate publisher reads the outbox:
Database
|
+-- orders
|
+-- outbox
|
v
Outbox Publisher
|
v
Message Broker
The publisher can retry publishing.
This makes the combination:
Saga
+
Transactional Outbox
particularly useful in event-driven architectures.
The Saga defines the workflow.
The Outbox helps reliably move the events that advance that workflow.
Message Delivery Is Not Perfect
Even with an outbox, distributed messaging introduces another reality:
Messages can be duplicated.
Messages can be delayed.
Messages can arrive out of order.
Consumers can crash.
Services can restart.
Networks can fail.
Therefore, a Saga must be designed around failure.
For example:
OrderCreated
↓
Payment Service
↓
PaymentCompleted
The Payment Service might process:
PaymentCompleted
but crash before acknowledging the message.
The broker delivers it again.
Now:
PaymentCompleted
↓
Payment Service
↓
PaymentCompleted
↓
Payment Service
The consumer must safely handle the duplicate.
This is why idempotency is not an optional optimization.
It is part of the correctness model.
Saga State
A complex Saga should usually have explicit state.
For example:
Saga ID: 8a3f...
Status:
STARTED
ORDER_CREATED
INVENTORY_RESERVED
PAYMENT_COMPLETED
SHIPMENT_CREATED
COMPLETED
Or on failure:
STARTED
ORDER_CREATED
INVENTORY_RESERVED
PAYMENT_COMPLETED
SHIPMENT_FAILED
COMPENSATING
PAYMENT_REFUNDED
INVENTORY_RELEASED
ORDER_CANCELLED
FAILED
The Saga state allows the system to answer:
Where is this Saga?
and:
What should happen next?
This becomes especially important when the workflow can take seconds, minutes, or even hours.
Long-Running Sagas
A traditional database transaction might run for:
10 ms
100 ms
1 second
A Saga can potentially run much longer.
For example:
Customer places order
↓
Payment authorization
↓
Fraud verification
↓
Warehouse confirmation
↓
Shipping provider confirmation
Some steps might depend on external systems.
The Saga might therefore remain active for:
seconds
minutes
hours
This is one reason Sagas are useful for long-running business processes.
But it also means you cannot keep database connections or locks open for the entire workflow.
Instead:
Local Transaction
↓
Commit
↓
Wait
↓
Next Local Transaction
↓
Commit
Each step remains short-lived.
Timeouts
Distributed workflows can get stuck.
For example:
Order Created
↓
Payment Service
↓
?????????
Perhaps the payment provider never responds.
The Saga cannot wait forever.
So we need timeouts:
Payment Requested
|
| wait
|
| 30 seconds
|
X
|
Payment Timeout
The Saga can then transition into:
COMPENSATING
and execute:
Cancel Order
Release Inventory
Timeouts therefore become part of the business workflow.
A timeout is not merely an infrastructure concern.
It can change the business state of the Saga.
Retries
Transient failures are common.
For example:
Payment Service
|
X
Connection timeout
The Saga may retry:
Attempt 1
↓
Timeout
↓
Attempt 2
↓
Timeout
↓
Attempt 3
↓
Success
But retries must be combined with idempotency.
Otherwise:
ChargePayment
↓
Timeout
↓
Retry
↓
ChargePayment again
could charge the customer twice.
This is why distributed workflows commonly require:
Retries
+
Idempotency
+
Timeouts
as a combined design.
Backoff
Retries should generally not happen immediately forever.
Instead:
Attempt 1
↓
100 ms
↓
Attempt 2
↓
500 ms
↓
Attempt 3
↓
2 seconds
↓
Attempt 4
This is exponential backoff.
The exact policy depends on the system.
The important principle is:
Retries should reduce the pressure on a failing dependency, not increase it.
A failing service receiving thousands of immediate retries can turn a temporary failure into a much larger outage.
Dead-Letter Handling
What happens if a message cannot be processed after several attempts?
For example:
OrderCreated
↓
Payment Service
↓
Failure
↓
Retry
↓
Failure
↓
Retry
↓
Failure
Eventually the message may need to be moved to a dead-letter mechanism:
Message Broker
|
+---- Normal Queue
|
+---- Dead Letter Queue
But a dead-letter queue is not the end of the Saga.
The system needs operational processes to determine:
Why did this Saga fail?
Should it be retried?
Should compensation run?
Does the business need manual intervention?
This is where observability becomes extremely important.
Observability for Sagas
A distributed Saga can involve:
10 services
20 messages
15 database transactions
5 retries
3 compensations
Debugging this without proper observability is extremely difficult.
Every Saga should have a correlation identifier:
sagaId = 8a3f2c...
That identifier should travel through:
HTTP requests
Events
Commands
Logs
Database records
Traces
Metrics
For example:
Saga ID: 8a3f2c
Order Service
|
| OrderCreated
|
Payment Service
|
| PaymentCompleted
|
Inventory Service
|
| InventoryFailed
|
Payment Service
|
| PaymentRefunded
|
Order Service
|
| OrderCancelled
Now the entire Saga can be reconstructed.
Distributed Tracing
Distributed tracing is particularly useful.
A trace might look like:
POST /orders
|
+-- Order Service
|
+-- Create Order
|
+-- Publish OrderCreated
|
+-- Payment Service
|
+-- Charge Payment
|
+-- Inventory Service
|
+-- Reserve Inventory
When something fails:
Inventory Service
|
X
Reserve failed
the trace can show the entire workflow.
This is much easier than searching individual service logs manually.
Metrics
Useful Saga metrics include:
Saga started
Saga completed
Saga failed
Saga compensated
Saga timed out
Saga duration
Compensation duration
Retry count
Dead-letter count
For example:
Saga Success Rate
Completed: 98.7%
Compensated: 1.1%
Timed out: 0.2%
Another useful metric:
Average Saga Duration
Order Saga
↓
450 ms
But also monitor:
p95
p99
because a small number of extremely slow Sagas can indicate serious problems even when the average looks healthy.
Saga State Machine
A useful way to model a Saga is as a state machine.
For example:
START
|
v
ORDER_CREATED
|
v
INVENTORY_RESERVED
|
v
PAYMENT_COMPLETED
|
v
SHIPMENT_CREATED
|
v
COMPLETED
Failure paths:
ORDER_CREATED
|
X
|
FAILED
or:
INVENTORY_RESERVED
|
X
|
COMPENSATING
|
v
ORDER_CANCELLED
|
v
FAILED
Or:
PAYMENT_COMPLETED
|
X
|
COMPENSATING
|
+----> REFUND_PAYMENT
|
+----> RELEASE_INVENTORY
|
+----> CANCEL_ORDER
|
v
FAILED
Thinking in terms of a state machine makes complex Saga behavior much easier to reason about.
What Happens if Compensation Fails?
This is one of the hardest Saga problems.
Suppose:
Charge Payment ✓
Reserve Inventory ✓
Create Shipment ✗
The Saga starts compensation:
Refund Payment
But the refund fails too.
Now:
Payment = CHARGED
Inventory = RESERVED
Shipment = NOT_CREATED
The Saga cannot simply say:
ROLLBACK
because there is no global rollback.
The compensation itself must be retried.
So we may have:
Shipment Failed
↓
Refund Payment
↓
Refund Failed
↓
Retry
↓
Refund Failed
↓
Retry
↓
Refund Successful
This means compensation is itself a distributed workflow.
This is an important insight:
Failure handling in a distributed system is itself distributed.
What If Compensation Is Impossible?
Sometimes compensation can fail permanently.
For example:
External Payment Provider
may be unavailable for an extended period.
The Saga might eventually reach:
COMPENSATION_FAILED
At that point the system may require:
Retry later
+
Alert
+
Manual intervention
For financial systems, this is particularly important.
You should never design a Saga assuming:
Compensation always succeeds.
Instead, design for:
Forward failure
+
Compensation failure
+
Retry
+
Recovery
+
Manual intervention when necessary
Saga and Two-Phase Commit
Saga is often discussed as an alternative to Two-Phase Commit.
Two-Phase Commit attempts to make multiple participants agree on one global commit.
Conceptually:
Coordinator
|
+---------+---------+
| | |
v v v
DB1 DB2 DB3
Phase 1:
"Can you commit?"
Phase 2:
"Commit."
Saga takes a different approach:
T1
↓
T2
↓
T3
↓
T4
If T4 fails:
C3
↓
C2
↓
C1
So:
2PC
|
+-- Global transaction
+-- Coordinated commit
+-- Stronger transactional semantics
+-- Potential blocking/coordination costs
Saga
|
+-- Local transactions
+-- Eventual consistency
+-- Explicit compensation
+-- Better suited to long-running workflows
The important point is not that Saga is "better" than 2PC in every situation.
They solve the problem differently.
Saga accepts a weaker consistency model in exchange for avoiding a global transaction coordinator and long-lived distributed locks.
Saga Does Not Mean "Eventually Everything Will Be Fine"
This is a common misunderstanding.
A Saga does not automatically guarantee consistency.
You still have to design:
Transaction boundaries
Compensations
Concurrency
Idempotency
Retries
Timeouts
Message delivery
Ordering
State transitions
Recovery
Observability
The Saga pattern gives you the structure.
The correctness still comes from your design.
Common Saga Mistakes
Mistake 1: Treating Compensation as Database Rollback
Wrong:
Saga compensation = rollback
Better:
Saga compensation = business operation that restores a valid business state
Mistake 2: Forgetting Idempotency
If:
ReserveInventory
is executed twice, the result must still be correct.
Every externally triggered Saga operation should be evaluated for duplicate execution.
Mistake 3: Assuming Messages Arrive Exactly Once
Do not build correctness around:
Exactly once
Assume:
At least once
+
duplicates
+
retries
and make consumers idempotent.
Mistake 4: Ignoring Compensation Failure
Do not assume:
Payment Refund → always succeeds
Compensation needs its own:
retry
timeout
monitoring
recovery
Mistake 5: Creating Huge Choreographies
If ten services participate in a Saga and dozens of events connect them, choreography can become extremely difficult to understand.
At that point, consider orchestration.
Mistake 6: Making the Orchestrator Too Smart
The opposite problem also exists.
The orchestrator should coordinate the workflow.
It should not necessarily contain all business logic from every service.
A healthy separation is:
Orchestrator
|
+-- Workflow coordination
|
+-- State transitions
|
+-- Compensation decisions
|
+-- Timeouts
|
+-- Retries
Services
|
+-- Domain logic
|
+-- Local transactions
|
+-- Local invariants
The orchestrator tells services what operation to perform.
The services decide how that operation is implemented.
Choosing Choreography vs Orchestration
A simple decision model is:
Simple workflow?
|
YES
|
v
Choreography may work
Complex workflow?
|
YES
|
v
Consider orchestration
More specifically:
Few services
Simple dependencies
Few compensation paths
Event-driven architecture
|
v
Choreography
Whereas:
Many services
Complex workflow
Many failure paths
Long-running process
Need centralized visibility
Complex compensation
|
v
Orchestration
There is no universal rule.
The important question is:
Where should the business workflow live?
In choreography:
Across events and services
In orchestration:
Inside the orchestrator
A Production-Oriented Saga Architecture
A realistic architecture might look like:
Client
|
v
Order Service
|
v
Saga Orchestrator
|
+-------------+-------------+
| | |
v v v
Inventory Payment Shipping
Service Service Service
| | |
v v v
Inventory DB Payment DB Shipping DB
| | |
+-------------+-------------+
|
v
Message Broker
|
v
Event Consumers
Each service uses:
Local Transaction
+
Transactional Outbox
+
Idempotent Consumer
The Saga layer provides:
Workflow State
+
Retries
+
Timeouts
+
Compensation
+
Recovery
And the platform provides:
Distributed Tracing
Metrics
Logs
Alerts
The complete picture becomes:
Client
|
v
API Gateway
|
v
Order Service
|
v
Saga Orchestrator
|
+-----------------+------------------+
| | |
v v v
Inventory Payment Shipping
Service Service Service
| | |
DB DB DB
| | |
+-----------------+------------------+
|
Outbox Events
|
v
Message Broker
|
v
Event Consumers
|
v
Observability Stack
This is the architecture you should have in mind when thinking about Sagas in production.
A Mental Model to Remember
Think of a Saga like organizing a large trip.
You need:
Book Flight
↓
Book Hotel
↓
Rent Car
There is no single transaction covering all three external systems.
If the car reservation fails:
Book Flight ✓
Book Hotel ✓
Rent Car ✗
you might:
Cancel Hotel
↓
Cancel Flight
You are not rolling back reality.
You are performing additional actions that compensate for what already happened.
That is exactly the idea behind a Saga.
The Core Saga Model
The complete model is:
Business Transaction
|
v
Saga Definition
|
+--------------+--------------+
| | |
v v v
T1 T2 T3
| | |
Local DB Local DB Local DB
| | |
+--------------+--------------+
|
Success
If something fails:
T1 ✓
|
T2 ✓
|
T3 ✗
|
v
Compensation
|
+----> C2
|
+----> C1
|
v
FAILED
The essential ingredients are:
1. Local transactions
2. Coordination
3. Explicit state
4. Compensating transactions
5. Idempotency
6. Retry handling
7. Timeout handling
8. Failure recovery
9. Event/message reliability
10. Observability
Putting Everything Together
A distributed business operation might look simple from the user's perspective:
"Place Order"
|
v
+-------------+
| Create Order|
+-------------+
|
v
+------------------+
| Reserve Inventory|
+------------------+
|
v
+---------------+
| Charge Payment|
+---------------+
|
v
+------------------+
| Create Shipment |
+------------------+
|
v
SUCCESS
But underneath, this is a distributed workflow:
Saga
|
+------------+------------+
| | |
v v v
Order Inventory Payment
Service Service Service
| | |
DB DB DB
| | |
+------------+------------+
|
v
Shipping
Service
|
DB
Each service has its own:
transaction boundary
database
failure modes
availability
latency
retry behavior
The Saga connects these independent systems into one business process.
But it does so without pretending that the entire distributed system is one database transaction.
That is the fundamental idea.
Final Architecture
A robust Saga implementation can be thought of as:
Business Request
|
v
Saga Started
|
v
+-------------------+
| Saga State Store |
+-------------------+
|
v
Saga Coordinator
|
+-----------------+-----------------+
| | |
v v v
Order Service Payment Service Inventory Service
| | |
DB DB DB
| | |
Outbox Outbox Outbox
| | |
+-----------------+-----------------+
|
v
Message Broker
|
v
Event Consumers
|
v
Next Saga Step
|
v
+----------+----------+
| |
Success Failure
| |
v v
COMPLETED COMPENSATING
|
+-----------+-----------+
| | |
v v v
Refund Release Cancel
Payment Inventory Order
| | |
+-----------+-----------+
|
v
COMPLETED
COMPENSATION
The most important thing to understand is that Saga is not a magic mechanism that gives microservices a distributed ROLLBACK.
It changes the problem.
Instead of asking:
How can I make several databases commit atomically?
we ask:
How can I divide this business operation into independently committed steps, and how can I safely recover if a later step fails?
That is a much more practical question for distributed systems.
Final Takeaway
The Saga Pattern is fundamentally about managing a business transaction that crosses service boundaries.
A Saga replaces:
One Global Transaction
with:
Local Transaction
↓
Local Transaction
↓
Local Transaction
↓
Local Transaction
and defines:
Compensation
for failure.
The complete mental model is:
Distributed Business Operation
|
v
Sequence of Steps
|
+-----------------+-----------------+
| | |
v v v
Local T1 Local T2 Local T3
| | |
+-----------------+-----------------+
|
Everything OK?
/ \
YES NO
| |
v v
COMPLETED COMPENSATE
|
+------+------+
| |
v v
C2 C1
| |
+------+------+
|
v
FAILED
And there are several principles worth remembering:
Saga
|
+-- No global ACID transaction
|
+-- Local transactions
|
+-- Eventual consistency
|
+-- Explicit compensation
|
+-- Choreography or orchestration
|
+-- Idempotent operations
|
+-- Retries and timeouts
|
+-- Reliable messaging
|
+-- Explicit Saga state
|
+-- Observability
|
+-- Recovery from compensation failure
The deepest lesson is this:
A Saga does not make a distributed system behave like a single database. It accepts that the system is distributed and explicitly models how the business process progresses and recovers across service boundaries.
Once you understand that, many other distributed-system patterns become easier to understand.
The Transactional Outbox Pattern explains how a service reliably publishes the events that advance a Saga.
Idempotency explains how Saga steps safely handle retries and duplicate messages.
Retry and Timeout Patterns explain how a Saga survives transient failures.
Circuit Breakers help prevent a failing dependency from taking down the workflow.
Distributed Tracing helps reconstruct what happened across the participating services.
And eventually, these patterns come together to form a practical toolkit for building reliable distributed systems.