Distributed systems fail differently from applications running entirely inside one process.
A local method call might look like:
service.calculatePrice()
If it fails, the failure is usually immediate.
A remote call is different:
Order Service
|
| HTTP
v
Pricing Service
The remote service might:
respond successfully
respond slowly
timeout
return 500
return 503
return 429
be completely unavailable
Now imagine the Pricing Service is down.
The Order Service keeps calling it.
Every request waits.
Threads become occupied.
Connections remain open.
Timeouts accumulate.
Traffic increases.
More resources are consumed.
Eventually the Order Service itself starts failing.
Pricing Service
X
|
v
slow / down
|
v
Order Service
|
waiting...
|
v
Thread Pool
|
v
exhausted
|
v
Order Service
X
The original failure was:
Pricing Service
But the final failure becomes:
Order Service
This is a cascading failure.
The Circuit Breaker pattern exists to prevent this situation.
The Core Problem
Consider:
pricingClient.getPrice(productId);
Normally:
Order Service
|
v
Pricing Service
|
v
Response
But when Pricing Service is unavailable:
Order Service
|
v
Pricing Service
X
The request may wait until a timeout:
Request
|
|---- 1s
|---- 2s
|---- 3s
|---- 4s
|---- timeout
|
v
Failure
Now imagine 1,000 concurrent requests.
1000 requests
|
v
Pricing Service
X
|
v
1000 waiting requests
|
v
Thread pool exhausted
The application can run out of:
threads
connections
memory
CPU
request slots
connection-pool capacity
The Circuit Breaker pattern prevents the application from continuously making calls that are very likely to fail.
What Is a Circuit Breaker?
A Circuit Breaker is a component placed between a caller and a remote dependency.
Instead of:
Order Service
|
v
Pricing Service
we have:
Order Service
|
v
Circuit Breaker
|
v
Pricing Service
The circuit breaker monitors calls.
When the dependency is healthy:
Order Service
|
v
Circuit Breaker
|
v
Pricing Service
When the dependency repeatedly fails:
Order Service
|
v
Circuit Breaker
X
|
v
Pricing Service
The circuit breaker stops sending requests.
The caller fails immediately instead of waiting for another timeout.
That is the fundamental idea.
Why Is It Called a Circuit Breaker?
The name comes from an electrical circuit breaker.
Normally:
Power
|
v
Circuit
|
v
Device
If a dangerous condition occurs:
Power
|
v
Circuit Breaker
X
|
v
Device
The circuit is interrupted.
A software circuit breaker does something conceptually similar:
Application
|
v
Circuit Breaker
X
|
v
Failing Dependency
The purpose is not to fix the dependency.
The purpose is to stop continuously sending traffic to it.
Circuit Breaker Is About Failure Containment
The most important mental model is:
Dependency failure
|
v
Circuit Breaker
|
v
Stop propagating the failure
Without a circuit breaker:
Dependency
|
X
|
v
Caller waits
|
v
Resources consumed
|
v
Caller slows down
|
v
Caller fails
|
v
Upstream callers fail
With a circuit breaker:
Dependency
|
X
|
v
Circuit opens
|
v
Requests fail fast
|
v
Caller remains healthy
Microsoft describes this as a self-preservation mechanism that can prevent a failing dependency from being overloaded further and can support graceful degradation.
The Three States
A typical circuit breaker has three states:
CLOSED
|
| failures exceed threshold
v
OPEN
|
| recovery timeout expires
v
HALF-OPEN
|
| success
v
CLOSED
The three states are:
Closed
Open
Half-Open
Understanding these states is the key to understanding the pattern.
CLOSED State
Initially the circuit is:
CLOSED
Requests are allowed through.
Client
|
v
Circuit Breaker
|
v
Service
The circuit breaker monitors the results.
For example:
Success
Success
Success
Success
Failure
Success
Success
The circuit remains closed.
A few failures don't necessarily mean the dependency is unavailable.
Failure Threshold
Suppose we configure:
failure threshold = 5
The breaker might behave like:
Failure 1
Failure 2
Failure 3
Failure 4
Failure 5
|
v
OPEN
But a production circuit breaker usually needs more nuance than simply counting five failures since startup.
For example, we might define:
5 failures
within 10 seconds
rather than:
5 failures
ever
This is important because failures from hours ago shouldn't normally cause the circuit to remain open forever.
Failure Rate
Instead of an absolute failure count, the breaker can use a failure rate.
For example:
minimum calls = 20
failure rate threshold = 50%
Suppose the last 20 calls contain:
12 failures
8 successes
Then:
failure rate = 60%
The circuit can open.
This is often more useful than simply counting failures because traffic volume varies.
Sliding Windows
Circuit breakers commonly evaluate recent traffic using a window.
For example:
last 20 requests
or:
last 10 seconds
Conceptually:
Time
------------------------------------------------>
[ recent calls ]
|
v
+---------------------+
| Success Success |
| Failure Failure |
| Failure Success |
+---------------------+
|
v
Failure rate
The window continuously moves forward.
Old results leave the window.
New results enter it.
OPEN State
Once the failure threshold is reached:
CLOSED
|
v
OPEN
Now requests are no longer sent to the dependency.
Instead:
Client
|
v
Circuit Breaker
|
X
|
v
Service
The dependency is not contacted.
The caller receives an immediate failure.
For example:
throw new CircuitBreakerOpenException();
This is called fail fast.
Why Fail Fast Matters
Suppose a dependency normally responds within:
100 ms
but is currently unreachable.
Without a circuit breaker:
Request
|
| 5 seconds
v
Timeout
With an open circuit:
Request
|
v
Circuit Breaker
|
v
Immediate failure
The difference is significant.
Instead of consuming resources for five seconds, the request can fail immediately.
Martin Fowler describes this as preventing callers from continuing to make remote calls that are likely to fail, which can otherwise lead to resource exhaustion and cascading failures.
HALF-OPEN State
An open circuit cannot stay open forever.
Eventually we need to determine:
Has the dependency recovered?
After a configured wait period:
OPEN
|
| wait
v
HALF-OPEN
The circuit allows a limited number of test requests.
For example:
HALF-OPEN
Request 1 → allowed
Request 2 → blocked
Request 3 → blocked
...
The first request acts as a probe.
Why Not Immediately Go to CLOSED?
Suppose Pricing Service crashed because it was overloaded.
After 30 seconds it starts recovering.
If the circuit immediately changes:
OPEN → CLOSED
then suddenly:
10,000 waiting requests
|
v
Pricing Service
The recovering service can be overwhelmed again.
Therefore:
OPEN
|
v
HALF-OPEN
|
| small number of requests
v
Service
The Half-Open state provides controlled recovery.
Microsoft specifically notes that allowing only limited requests during recovery prevents a recovering dependency from being flooded.
Successful Recovery
Suppose the circuit is:
HALF-OPEN
and the test requests succeed:
Probe 1 → success
Probe 2 → success
Probe 3 → success
The breaker concludes:
Dependency recovered
and transitions:
HALF-OPEN
|
v
CLOSED
Normal traffic resumes.
Failed Recovery
Suppose:
HALF-OPEN
and the test request fails:
Probe 1 → failure
The dependency is probably still unhealthy.
The breaker transitions:
HALF-OPEN
|
v
OPEN
The recovery timer starts again.
Complete State Machine
The entire lifecycle looks like:
failures exceed threshold
+----------------------------+
| |
v |
+--------+ |
| CLOSED | |
+--------+ |
| |
| |
v |
+--------+ |
| OPEN |<----------------------+
+--------+
|
| recovery timeout
v
+-----------+
| HALF-OPEN |
+-----------+
| |
success failure
| |
v |
CLOSED <---+
This state machine is the heart of the Circuit Breaker pattern.
Circuit Breaker vs Retry
These patterns are often confused.
They solve different problems.
Retry
Retry says:
The operation failed.
It might succeed if I try again.
For example:
Request
|
X
|
v
wait
|
v
retry
|
v
success
Retry is useful for transient failures.
Circuit Breaker
Circuit Breaker says:
This dependency is probably unhealthy.
Stop calling it for now.
So:
Retry
|
v
Try again
while:
Circuit Breaker
|
v
Stop trying
Microsoft explicitly distinguishes the two: retry attempts to recover from transient failures, while a circuit breaker prevents calls when the dependency is likely to remain unavailable.
Retry + Circuit Breaker
They are often used together.
Consider:
Request
|
v
+----------------+
| Circuit Breaker|
+----------------+
|
v
Retry
|
v
Remote Service
A transient failure:
Request
|
v
Failure
|
v
Retry
|
v
Success
A persistent failure:
Request
|
v
Failure
|
v
Retry
|
v
Failure
|
v
Failure threshold
|
v
Circuit OPEN
After the circuit opens:
Future requests
|
v
Circuit Breaker
|
X
They don't even reach the retry mechanism.
Why Retry Alone Is Dangerous
Imagine:
Order Service
|
v
Payment Service
X
The Order Service retries each request three times.
Now there are:
100 requests
×
3 retries
=
300 calls
If the Payment Service is already overloaded, retries can make the problem worse.
This is a retry storm.
Now imagine every service does the same thing:
Service A
|
v
Service B
|
v
Service C
If C fails:
A retries B
B retries C
Traffic can multiply rapidly.
Circuit breakers help stop persistent failures from continuously consuming resources.
Retry Backoff
Retries should normally use backoff.
Instead of:
retry immediately
retry immediately
retry immediately
use:
attempt 1
|
| 100ms
v
attempt 2
|
| 200ms
v
attempt 3
|
| 400ms
v
attempt 4
Usually exponential backoff plus jitter is preferable.
But even good retry behavior cannot replace a circuit breaker when a dependency remains unhealthy.
Circuit Breaker vs Timeout
Timeout and Circuit Breaker are also different.
A timeout says:
Don't wait longer than X.
For example:
HTTP timeout = 2 seconds
Circuit breaker says:
Don't make the call at all because recent evidence suggests it will fail.
They work together.
Circuit Breaker
|
v
Timeout
|
v
Remote Service
A timeout detects an individual slow call.
A circuit breaker reacts to a pattern of failures across calls.
Why Timeout Is Essential
A circuit breaker cannot prevent the first few calls from being slow.
Suppose:
Circuit = CLOSED
and the dependency hangs.
Without a timeout:
Request
|
v
Dependency
|
|----------------------+
|
waiting
The circuit breaker may not know the call failed until the underlying operation returns.
Therefore:
Timeout
+
Circuit Breaker
is a much stronger combination.
Microsoft specifically warns that overly long downstream timeouts can still tie up threads and other resources even when a circuit breaker is present.
Circuit Breaker vs Bulkhead
Bulkhead is another resilience pattern that is frequently used with circuit breakers.
A circuit breaker says:
Stop calling the unhealthy dependency.
A bulkhead says:
Don't let one dependency consume all available resources.
Without bulkheads:
Application
|
+---------+---------+
| |
v v
Payment Shipping
| |
thread pool thread pool
If Payment becomes slow, it might consume all threads.
With bulkheads:
Application
|
+---- Payment Pool
|
+---- Shipping Pool
Payment cannot consume Shipping's resources.
Microsoft recommends combining bulkheads with retry and circuit breakers for stronger failure isolation.
The Three Patterns Together
A robust remote call might look like:
Request
|
v
+-------------+
| Bulkhead |
+-------------+
|
v
+-------------+
| Circuit |
| Breaker |
+-------------+
|
v
Timeout
|
v
Retry
|
v
Remote Service
Each solves a different problem:
Bulkhead
limits resource consumption
Circuit Breaker
stops calls to unhealthy dependencies
Timeout
limits waiting time
Retry
handles transient failures
These patterns complement each other rather than replacing one another.
What Should Trip the Circuit?
Not every error should open the circuit.
This is one of the most important implementation decisions.
Suppose the dependency returns:
400 Bad Request
That usually means:
The caller sent invalid data.
Retrying won't fix it.
And opening the circuit is usually wrong.
Consider:
401 Unauthorized
Again, this isn't normally a dependency availability problem.
Now consider:
500 Internal Server Error
503 Service Unavailable
timeout
connection refused
connection reset
These may indicate dependency failure.
The breaker should therefore distinguish:
business/client failures
from:
dependency failures
HTTP Status Codes
A simplistic circuit breaker might count every non-2xx response as a failure.
That is usually incorrect.
For example:
400
401
403
404
may be valid application outcomes.
Depending on the system, you might count:
500
502
503
504
timeouts
connection failures
as circuit-breaker failures.
And perhaps:
429 Too Many Requests
depending on the dependency's semantics and the retry policy.
The exact policy should be based on what the error means, not simply whether HTTP status is >= 400.
Business Errors vs Infrastructure Errors
Consider:
Payment Service
It returns:
402 Payment Required
because the customer's card was declined.
This is a business outcome.
The Payment Service is healthy.
Opening the circuit would be wrong.
Compare:
Payment Service
|
X
Connection refused
This is an infrastructure failure.
The circuit should potentially count it.
The distinction is:
Business failure
≠
Dependency failure
This is critical.
Circuit Breaker Scope
Suppose your application calls:
Payment Service
Inventory Service
Shipping Service
Should there be one global circuit breaker?
Usually not.
You generally want independent circuit state per protected dependency or meaningful resource boundary.
Application
|
+-----------+-----------+
| | |
v v v
Payment CB Inventory CB Shipping CB
| | |
Payment Inventory Shipping
If Payment is down:
Payment Circuit = OPEN
but:
Inventory Circuit = CLOSED
Shipping Circuit = CLOSED
The Payment failure should not automatically block unrelated dependencies.
Circuit Breaker Per Dependency
Consider:
Order Service
with:
Pricing Service
Payment Service
Inventory Service
A good model is:
Order Service
Pricing
|
+-- Circuit A
Payment
|
+-- Circuit B
Inventory
|
+-- Circuit C
Now:
Pricing failure
does not cause:
Payment circuit → OPEN
This limits the blast radius.
Circuit Breaker Per Endpoint
Sometimes even a single service needs finer isolation.
For example:
Customer Service
GET /customers/{id}
POST /customers
GET /customers/{id}/orders
If one operation is particularly unreliable, you may need independent protection.
However, overly granular circuit breakers can create operational complexity.
The right boundary depends on:
failure behavior
resource sharing
traffic patterns
business impact
Local vs Distributed Circuit Breakers
A circuit breaker is usually local to the caller.
Suppose:
Order Service
has three instances:
Order-1
Order-2
Order-3
Each may maintain its own circuit state:
Order-1 → Payment = OPEN
Order-2 → Payment = CLOSED
Order-3 → Payment = OPEN
This is often desirable because circuit state is based on the traffic each instance observes.
But it also means that recovery traffic can vary between instances.
Should Circuit State Be Shared?
You could create:
Shared Circuit State
using Redis or another distributed mechanism.
But now the circuit breaker itself becomes a distributed system component.
You introduce:
network calls
shared state
consistency concerns
failure modes
latency
For many applications, a local circuit breaker is simpler and sufficient.
Distributed circuit state should have a clear reason to exist.
Fallbacks
Opening a circuit does not automatically mean:
return HTTP 500
Sometimes the application can degrade gracefully.
For example:
Recommendation Service
|
X
|
v
Circuit OPEN
|
v
Return cached recommendations
Or:
Pricing Service unavailable
|
v
Show "Price temporarily unavailable"
Or:
Shipping Service unavailable
|
v
Allow order creation
but delay shipping calculation
The correct fallback is a business decision.
Graceful Degradation
Suppose an e-commerce homepage depends on:
Product Service
Recommendation Service
Review Service
Recommendation Service goes down.
Instead of:
Entire homepage → 500
the system can return:
Product data → available
Reviews → available
Recommendations → unavailable
The user sees:
Homepage
---------------------
Products
Reviews
Recommendations
temporarily unavailable
---------------------
The system has degraded gracefully.
Circuit breakers can trigger this behavior. Microsoft explicitly identifies graceful degradation as one of the uses of the pattern.
Fallback Must Not Hide Serious Failures
A fallback isn't always appropriate.
For example:
Payment Service unavailable
You probably should not silently use:
fake payment success
Instead:
Payment temporarily unavailable.
Please try again.
Fallbacks must preserve business correctness.
Cache as a Fallback
Caching is a common fallback.
For example:
Client
|
v
Product Service
|
X
|
v
Circuit Breaker
|
v
Cache
If the data is acceptable when slightly stale:
return cached data
This is especially useful for:
configuration
catalogs
recommendations
exchange rates
metadata
feature flags
But stale data can be dangerous for:
account balance
inventory availability
payment status
authorization
The fallback must match the consistency requirements.
Circuit Breaker and Idempotency
Suppose a payment request is sent:
POST /payments
The Payment Service processes it successfully.
But the response is lost:
Payment Service
|
v
Payment completed
|
X
response lost
The caller sees:
timeout
The circuit breaker may count this as a failure.
But retrying the payment can create a duplicate charge.
Therefore:
Circuit Breaker
+
Retry
+
Idempotency
often need to be considered together.
The circuit breaker does not make an operation safe to retry.
Circuit Breaker Does Not Solve Duplicate Requests
Suppose:
Payment request
times out.
The circuit breaker says:
The dependency appears unhealthy.
It does not know whether the payment was actually processed.
Therefore:
Circuit Breaker
solves:
dependency failure propagation
while:
Idempotency
solves:
duplicate operation effects
Different problems.
Circuit Breaker and Saga
Consider an order workflow:
Order
|
v
Payment
|
v
Inventory
|
v
Shipping
Suppose Inventory Service is unavailable.
The circuit breaker may immediately reject the inventory call:
Inventory Circuit
|
X
The Saga can then decide:
Order cannot continue
and execute compensation:
Refund Payment
Cancel Order
So:
Circuit Breaker
|
+-- protects individual remote calls
Saga
|
+-- coordinates the business workflow
The circuit breaker does not replace the Saga.
Circuit Breaker and Queue-Based Systems
Circuit breakers are particularly natural for synchronous calls.
For asynchronous systems:
Producer
|
v
Queue
|
v
Consumer
other mechanisms may be more appropriate:
retry
dead-letter queue
backoff
visibility timeout
poison-message handling
A queue already provides buffering and decoupling.
Therefore blindly putting circuit breakers everywhere is unnecessary.
Microsoft notes that message-driven architectures often have their own retry and dead-letter mechanisms and may not need a circuit breaker in the same way synchronous request/response systems do.
Circuit Breaker and Service Mesh
Circuit breaking can also be implemented outside application code.
For example:
Application
|
v
Service Mesh
|
v
Remote Service
The service mesh can provide:
timeouts
retries
connection limits
outlier detection
circuit breaking
This can centralize resilience behavior.
But application-level circuit breakers still have an advantage:
The application understands business semantics.
For example:
Payment declined
is different from:
Payment Service unavailable
Infrastructure cannot always make that distinction.
Application-Level vs Infrastructure-Level
Application-level:
Order Service
|
v
Circuit Breaker
|
v
Payment Service
Infrastructure-level:
Order Service
|
v
Proxy / Service Mesh
|
v
Payment Service
Both can be useful.
The important thing is to avoid accidentally stacking multiple independent retry and circuit-breaking policies without understanding their combined behavior.
For example:
Application Retry
+
Application Circuit Breaker
+
Service Mesh Retry
+
Service Mesh Circuit Breaker
can create surprising traffic amplification and recovery behavior.
A Simple Java Implementation
A conceptual circuit breaker can be implemented as:
public class CircuitBreaker {
private State state = State.CLOSED;
public <T> T execute(Supplier<T> operation) {
if (state == State.OPEN) {
throw new CircuitOpenException();
}
try {
T result = operation.get();
recordSuccess();
return result;
} catch (Exception e) {
recordFailure(e);
throw e;
}
}
}
But this is only the beginning.
A real implementation needs:
failure counting
sliding windows
state transitions
timers
concurrency control
half-open probing
metrics
thread safety
exception classification
This is why production applications normally use a mature resilience library rather than implementing the entire mechanism themselves.
Spring Boot
A common Java/Spring approach is to use a resilience library such as Resilience4j.
Conceptually:
@CircuitBreaker(
name = "paymentService",
fallbackMethod = "paymentFallback"
)
public PaymentResult pay(PaymentRequest request) {
return paymentClient.pay(request);
}
The architecture becomes:
Controller
|
v
Service
|
v
Circuit Breaker
|
v
Payment Client
|
v
Payment Service
The important part isn't the annotation.
The important part is understanding what policy the annotation represents.
Configuration
A circuit breaker might conceptually have:
circuitBreaker:
failureRateThreshold: 50
minimumNumberOfCalls: 20
slidingWindowSize: 20
waitDurationInOpenState: 10s
permittedCallsInHalfOpenState: 3
Meaning:
minimum calls = 20
failure rate = 50%
open wait = 10 seconds
half-open probes = 3
The exact configuration should be based on the dependency and workload.
There is no universally correct:
threshold = 5
timeout = 10s
Why Configuration Matters
Consider:
failure threshold = 1
One temporary network packet loss could open the circuit.
That's too sensitive.
Now consider:
failure threshold = 1000
The dependency may be effectively dead before the circuit reacts.
That's too slow.
The breaker needs enough evidence to distinguish:
occasional transient failures
from:
persistent dependency failure
Failure Rate vs Slow Calls
Not every dangerous dependency returns errors.
Consider:
Payment Service
which normally responds in:
100 ms
but suddenly takes:
10 seconds
Technically:
HTTP 200
The requests are successful.
But the system may still be unhealthy because threads and connections are being held for too long.
Therefore resilience policies may also consider:
slow-call rate
in addition to:
failure rate
This is particularly important for preventing resource exhaustion caused by slow dependencies.
Slow Dependency Can Be Worse Than a Failed Dependency
A completely dead service might produce:
connection refused
immediately.
That's actually easy to handle.
A degraded service might produce:
response after 30 seconds
That is often more dangerous.
Consider:
1000 requests
×
30 seconds
Many resources remain occupied.
This is why:
Timeout
+
Circuit Breaker
is important.
Circuit Breaker and Latency
A healthy system:
p99 = 200ms
Dependency degrades:
p99 = 5s
Then:
p99 = 10s
Even before errors become common, the application may be approaching a resource exhaustion problem.
Therefore circuit-breaker policies should consider both:
failure
and, where supported:
slow calls
Monitoring
A circuit breaker without observability is dangerous.
At minimum, monitor:
circuit state
failure rate
success rate
slow-call rate
rejected calls
open transitions
half-open transitions
closed transitions
fallback count
dependency latency
dependency error rate
For example:
paymentService.circuit.state = OPEN
paymentService.circuit.failureRate = 82%
paymentService.circuit.rejectedCalls = 14,392
This immediately tells an operator:
Payment dependency is unhealthy
State Transition Metrics
Track transitions:
CLOSED → OPEN
OPEN → HALF_OPEN
HALF_OPEN → CLOSED
HALF_OPEN → OPEN
A circuit repeatedly cycling:
CLOSED
↓
OPEN
↓
HALF_OPEN
↓
CLOSED
↓
OPEN
↓
HALF_OPEN
↓
CLOSED
may indicate an unstable dependency.
This is often called flapping.
Logging
When the circuit opens, log it.
For example:
Circuit opened for paymentService
failureRate=78%
window=20 calls
threshold=50%
When it recovers:
Circuit closed for paymentService
successfulHalfOpenCalls=3
But avoid logging every rejected request at error level.
If 100,000 requests hit an open circuit:
100,000 error logs
can create another operational problem.
Log state transitions prominently and use metrics for volume.
Distributed Tracing
A request might travel:
Client
|
v
API Gateway
|
v
Order Service
|
v
Payment Service
|
v
Banking API
If Payment Service fails, tracing helps identify where the latency and failure originated.
Circuit-breaker events should be correlated with:
trace ID
service
dependency
operation
circuit state
exception
latency
Microsoft recommends distributed tracing and clear observability when implementing circuit breakers.
Testing Circuit Breakers
Don't only test:
Service works
Test failure scenarios.
Dependency returns 500
Payment → 500
Verify:
failure recorded
Dependency times out
Payment → timeout
Verify:
timeout recorded
Threshold reached
failure
failure
failure
...
Verify:
CLOSED → OPEN
Open circuit
Verify:
request does not reach dependency
Recovery
Verify:
OPEN → HALF_OPEN
Successful probe
Verify:
HALF_OPEN → CLOSED
Failed probe
Verify:
HALF_OPEN → OPEN
Fallback
Verify:
OPEN → expected degraded behavior
A Critical Test
One particularly important test is:
When the circuit is OPEN,
does the request actually avoid the dependency?
For example:
Circuit OPEN
|
v
GET /payment
should result in:
Payment Service calls = 0
If the application still calls the dependency, the circuit breaker isn't actually protecting it.
Common Mistake: Opening the Circuit Too Quickly
Suppose:
threshold = 2 failures
A temporary network problem causes:
Failure
Failure
Circuit opens.
But the dependency was actually healthy.
The application now unnecessarily fails requests.
A circuit breaker must tolerate some transient failures.
Common Mistake: Opening Too Slowly
The opposite problem:
threshold = 1000 failures
The dependency fails.
The application continues sending requests.
By the time the circuit opens:
threads exhausted
connection pool exhausted
latency increased
upstream services failing
The circuit breaker reacted too late.
Common Mistake: No Timeout
This is particularly dangerous.
You configure:
Circuit Breaker
but don't configure:
Timeout
The dependency hangs.
The circuit breaker waits for the call to finish.
Resources remain occupied.
A circuit breaker is not a replacement for a timeout.
Common Mistake: Retrying When the Circuit Is Open
Suppose:
Circuit = OPEN
and the caller's retry logic says:
try again
try again
try again
That defeats the purpose.
The retry mechanism should understand:
CircuitOpenException
as:
Do not retry immediately.
Microsoft explicitly recommends that retry logic be aware of circuit-breaker responses and stop retrying when the breaker indicates a persistent fault.
Common Mistake: Retrying Non-Idempotent Operations
Consider:
POST /payments
The request might succeed but the response may be lost.
Retrying could create:
Payment #1
Payment #2
Therefore:
Retry
must be combined with:
Idempotency
for operations where duplicate effects are dangerous.
Common Mistake: Using One Circuit for Everything
Don't do:
Application
|
v
Global Circuit Breaker
|
+---- Payment
+---- Inventory
+---- Shipping
If Inventory fails:
Everything blocked
Instead, normally isolate dependencies:
Payment Circuit
Inventory Circuit
Shipping Circuit
so one dependency's failure doesn't unnecessarily affect unrelated operations.
Common Mistake: Treating Business Errors as Dependency Failures
Do not automatically count:
400
401
403
404
as infrastructure failures.
For example:
Payment declined
does not mean:
Payment Service is down.
Circuit breakers need semantic failure classification.
Common Mistake: Bad Fallbacks
A fallback should not silently change business correctness.
Bad:
Payment unavailable
|
v
pretend payment succeeded
Good:
Payment unavailable
|
v
return temporary-unavailability response
or, if business rules permit:
Order created
Payment pending
The fallback must preserve domain correctness.
Common Mistake: Ignoring Recovery Behavior
Opening a circuit is easy.
Recovery is harder.
Suppose:
OPEN → CLOSED
too quickly.
Traffic can overwhelm the recovering dependency.
Suppose:
OPEN → HALF_OPEN
too slowly.
The application may remain degraded even after the dependency has recovered.
Recovery timing must match the dependency's behavior. Microsoft explicitly calls recoverability and the open-state duration important circuit-breaker configuration considerations.
Circuit Breaker as a State Machine
A useful way to reason about the pattern is:
healthy
|
v
+---------+
| CLOSED |
+---------+
|
persistent failure
|
v
+---------+
| OPEN |
+---------+
|
wait / cooldown
|
v
+---------+
| HALF- |
| OPEN |
+---------+
/ \
success failure
/ \
v v
+---------+ +---------+
| CLOSED | | OPEN |
+---------+ +---------+
Thinking of the breaker as a state machine makes implementation and debugging much easier.
The Complete Remote Call
A mature synchronous remote call can look like:
Request
|
v
Bulkhead
|
v
Circuit Breaker
|
circuit closed?
/ \
no yes
| |
v v
Fail Fast Timeout
|
v
Retry
|
v
Remote Service
|
+----------+----------+
| |
success failure
| |
v v
Return record failure
|
v
update circuit state
Notice how each mechanism has a specific responsibility.
A Practical Example
Suppose an Order Service calls:
POST /payments
Configuration:
timeout = 2 seconds
retry:
maximum attempts = 2
exponential backoff
circuit breaker:
minimum calls = 20
failure threshold = 50%
open duration = 10 seconds
half-open probes = 3
Normal operation:
Order
|
v
Circuit CLOSED
|
v
Payment
|
v
Success
Temporary failure:
Payment
|
X
|
v
Retry
|
v
Payment
|
v
Success
Persistent failure:
Payment
|
X
|
v
Retry
|
X
|
v
Failure rate > 50%
|
v
Circuit OPEN
Subsequent requests:
Request
|
v
Circuit OPEN
|
X
Payment
After 10 seconds:
OPEN
|
v
HALF-OPEN
Probe:
Probe
|
v
Payment
|
v
Success
After enough successful probes:
HALF-OPEN
|
v
CLOSED
Normal traffic resumes.
Circuit Breaker Does Not Repair the Dependency
This is an important conceptual distinction.
The circuit breaker doesn't fix:
Payment Service
It only changes the behavior of:
Order Service
from:
keep calling
to:
stop calling temporarily
The actual dependency may need:
restart
scaling
bug fix
database recovery
network recovery
capacity increase
The circuit breaker simply gives it time to recover without continuously receiving more traffic.
Circuit Breaker as Backpressure
At a high level, the circuit breaker creates a form of backpressure.
Without it:
Dependency failing
^
|
| continuous requests
|
Caller ---------------->
With it:
Dependency failing
Caller
|
v
Circuit Breaker
|
X
Traffic is cut off.
This protects both:
caller
and:
dependency
from additional damage.
Circuit Breaker and Load Shedding
An open circuit is also a form of load shedding.
Instead of allowing every request to reach an unhealthy dependency:
10,000 requests
|
v
Dependency
we may do:
10,000 requests
|
v
Circuit Breaker
|
+---- 9,990 rejected/fallback
|
+---- 10 controlled probes
The dependency gets an opportunity to recover.
Circuit Breaker and Graceful Degradation
These ideas work particularly well together.
Dependency fails
|
v
Circuit opens
|
v
Fallback
|
v
Reduced functionality
For example:
Recommendation Service
X
|
v
Circuit OPEN
|
v
Show popular products
The application remains useful even though one dependency is unavailable.
Circuit Breaker and Availability
Suppose:
Payment Service availability = 99.9%
Your application depends synchronously on it.
Without resilience:
Payment unavailable
|
v
Order unavailable
With graceful degradation:
Payment unavailable
|
v
Payment functionality unavailable
|
v
Other Order functionality remains available
The goal is not necessarily:
Everything always works.
The goal is:
One failure should not take down everything.
Production Architecture
A realistic architecture might look like:
Client
|
v
API Gateway
|
v
Order Service
|
+----------------+----------------+
| | |
v v v
Pricing CB Payment CB Inventory CB
| | |
v v v
Pricing Payment Inventory
Service Service Service
| | |
v v v
Database Database Database
Each dependency has independent protection.
Now suppose:
Payment Service
X
Only:
Payment Circuit
opens.
Pricing and Inventory continue functioning.
Failure Isolation
This is the bigger architectural principle behind Circuit Breaker.
Without isolation:
Payment failure
|
v
Order failure
|
v
Checkout failure
|
v
API failure
With isolation:
Payment failure
|
v
Payment Circuit OPEN
|
v
Payment functionality degraded
Pricing ────────────────> works
Inventory ──────────────> works
Catalog ────────────────> works
The failure remains contained.
How Circuit Breaker Fits Into the Resilience Toolkit
A production distributed system typically needs several complementary mechanisms:
Resilience
|
+-------------+-------------+
| | |
Timeout Retry Circuit
| | Breaker
| | |
+-------------+-------------+
|
Bulkhead
|
Fallback
|
Observability
Each solves a different failure mode.
Timeout
Don't wait forever.
Retry
Try transient failures again.
Circuit Breaker
Stop persistent failures.
Bulkhead
Limit resource blast radius.
Fallback
Degrade gracefully.
Idempotency
Make retries safe.
Observability
Tell us what is happening.
Final Architecture
The complete mental model is:
Incoming Request
|
v
+-------------+
| Bulkhead |
+-------------+
|
v
+-------------------+
| Circuit Breaker |
+-------------------+
| |
OPEN | | CLOSED
| |
v v
Fallback Timeout
|
v
Retry
|
v
Remote Service
|
+---------+---------+
| |
Success Failure
| |
v v
Return Record Failure
|
v
Update Circuit
This architecture protects the application from:
slow dependencies
failed dependencies
retry storms
resource exhaustion
cascading failures
while still allowing automatic recovery.
Final Takeaway
The Circuit Breaker pattern is fundamentally about one idea:
Don't keep calling something that is clearly failing.
Without a circuit breaker:
Dependency fails
|
v
Keep calling
|
v
Keep waiting
|
v
Consume resources
|
v
Caller fails
|
v
Cascading failure
With a circuit breaker:
Dependency fails
|
v
Detect repeated failures
|
v
OPEN
|
v
Fail fast
|
v
Dependency gets time to recover
|
v
HALF-OPEN
|
v
Controlled probes
|
v
CLOSED
The three states are the core:
CLOSED
normal operation
OPEN
reject calls immediately
HALF-OPEN
cautiously test recovery
But a production-quality implementation requires much more than those three states.
You need to think about:
timeouts
failure classification
failure thresholds
failure rates
sliding windows
slow calls
retries
backoff
idempotency
fallbacks
bulkheads
concurrency
recovery
observability
Most importantly, a Circuit Breaker should not be viewed as an isolated library feature.
It is part of a larger resilience strategy:
Timeout
+
Retry
+
Circuit Breaker
+
Bulkhead
+
Fallback
+
Idempotency
+
Observability
The goal of these patterns is not to make failures disappear.
Distributed systems will fail.
The goal is to make sure:
one dependency failing
does not become:
the entire system failing.
That is the real purpose of the Circuit Breaker pattern.