A distributed system is often introduced with a simple definition:

A distributed system is a system where multiple independent computers work together and appear to the user as a single system.

That definition is correct.

But it does not explain why distributed systems are difficult.

Writing code on one machine is relatively straightforward. A function is called, memory is shared within the process, the disk either returns a result or an error, and the operating system provides a reasonably clear execution environment.

The moment we split that system across multiple machines, many assumptions stop being true.

A remote service might be slow.

It might be down.

The network might lose the request.

The request might succeed, but the response might be lost.

A retry might execute the same operation twice.

Two replicas might temporarily disagree.

Messages might arrive out of order.

One part of a distributed transaction might succeed while another fails.

And sometimes the most difficult situation is this:

You do not know what happened.

This is the fundamental challenge of distributed systems.

The problem is not simply that multiple machines exist.

The problem is that those machines communicate through an unreliable network, operate independently, observe time differently, and can fail independently.

This article covers the fundamental concepts that every backend developer should understand before designing distributed systems.

We will look at:

CAP theorem
Consistency models
Strong vs eventual consistency
Distributed transactions
Network failures
Partial failures
Timeouts
Retries
Idempotency
Ordering
Deduplication
Backpressure
Load balancing
Service discovery

The goal is not just to define these terms.

The goal is to understand why they exist and how they influence real system design.

1. The Fundamental Problem: A Remote Call Is Not a Function Call

Consider a monolithic application:

Order order = orderService.createOrder(request);

This looks simple.

The method is called.

The code executes.

A result or exception is returned.

Now consider a distributed version:

Order Service
      |
      | HTTP / gRPC
      v
Payment Service

The code might look similar:

PaymentResult result = paymentClient.charge(request);

But this operation is fundamentally different.

The call now involves multiple steps:

1. Serialize request
2. Send request to network
3. Network routes request
4. Payment service receives request
5. Payment service processes request
6. Payment service writes to database
7. Payment service sends response
8. Response travels through network
9. Order service receives response

Any of these steps can fail.

For example:

Order Service
      |
      | ------ Charge $100 ------> Payment Service
      |
      | <----- Response lost ---- X
      |
Timeout

Now the Order Service has a problem.

Did the payment succeed?

There are several possibilities:

A. Request never reached Payment Service
B. Payment Service received it but failed
C. Payment succeeded, but response was lost
D. Payment succeeded, but Order Service timed out before receiving the response

From the Order Service's perspective, these situations may look identical.

This is one of the most important ideas in distributed systems:

A timeout does not tell you whether the remote operation failed.

It only tells you that you did not receive a response in time.

Everything else in this article follows from this fundamental reality.

2. CAP Theorem

CAP theorem is one of the most frequently mentioned and most frequently misunderstood concepts in distributed systems.

CAP stands for:

Consistency
Availability
Partition Tolerance

The theorem states that when a network partition occurs, a distributed system must choose between consistency and availability.

Let's understand each term.

Consistency

In CAP, consistency means that every read receives the most recent successful write or an error.

Suppose we have two replicas:

Client
   |
   v
+---------+        +---------+
| Node A  | <----> | Node B  |
+---------+        +---------+

A client writes:

balance = 100

After that write succeeds, another client reading from either node should not see an older value.

That is CAP consistency.

Availability

Availability means that every request to a non-failed node receives a response.

The response does not necessarily have to contain the latest data.

For example:

Client ---> Node B

Node B: "I cannot contact Node A,
         but I will still respond with the data I have."

The system remains available.

But the data may be stale.

Partition Tolerance

A network partition occurs when nodes cannot communicate with each other.

Imagine:

        Network Partition

+---------+       X       +---------+
| Node A  | <------------- | Node B  |
+---------+               +---------+

Both nodes may still be running.

Neither machine has necessarily crashed.

But communication between them has failed.

This is the important part of CAP:

In a distributed system, you cannot simply decide that network partitions will never happen.

If your application runs across multiple machines, networks can fail.

Therefore, when a partition occurs, you have two major choices.

Choose consistency

Node B cannot verify the latest state.

Node B:
"I will reject or delay the request."

The system may become temporarily unavailable, but it avoids returning potentially inconsistent data.

Choose availability

Node B:
"I cannot reach the other nodes,
but I will continue responding."

The system remains available, but different nodes may temporarily return different values.

So CAP is better understood as:

During a network partition:

Consistency
      OR
Availability

Partition tolerance is not usually an optional third choice in a real distributed system.

The practical question is:

What should the system do when machines cannot communicate?

3. Consistency Models

Consistency is not a single concept.

Distributed systems can provide different guarantees about what values clients are allowed to observe.

Think of consistency models as rules about the visibility and ordering of data changes.

Strong Consistency

Suppose we write:

X = 10

After the write succeeds:

Read X -> 10

Every subsequent read should observe the latest value.

Conceptually:

Write
  |
  v
X = 10
  |
  v
All future reads see 10

Strong consistency makes the distributed system easier to reason about because it behaves more like a single system.

But providing strong consistency often requires coordination between nodes.

Coordination costs latency and can reduce availability during failures.

Eventual Consistency

With eventual consistency, replicas may temporarily disagree.

For example:

Initial state

Node A: X = 1
Node B: X = 1
Node C: X = 1

A write occurs:

X = 2

Immediately afterward:

Node A: X = 2
Node B: X = 1
Node C: X = 1

Later:

Node A: X = 2
Node B: X = 2
Node C: X = 2

Eventually, the replicas converge.

The important word is eventually.

Eventual consistency does not mean:

Data is randomly inconsistent forever.

It means that if updates stop and communication eventually succeeds, replicas should converge.

This is often acceptable for data such as:

View counts
Analytics
Search indexes
Social media feeds
Product catalogs
Recommendations

But it may be dangerous for:

Bank balances
Inventory reservations
Unique identifiers
Critical account state

The correct consistency model depends on what the data represents.

Read-Your-Writes Consistency

Consider this scenario.

A user changes their profile name:

User -> "Krrish"

The write succeeds.

Immediately afterward:

GET /profile

But the request goes to another replica that has not yet received the update.

The user sees:

Old Name

This can be surprising.

Read-your-writes consistency guarantees:

After you successfully write something, your subsequent reads should observe that write.

Other users may still temporarily see older data.

This is a useful example of how consistency can be defined from the perspective of an individual client.

4. Strong Consistency vs Eventual Consistency

The choice is not:

Strong consistency = good
Eventual consistency = bad

The real question is:

What happens if this particular data is temporarily stale?

Consider an e-commerce system.

Product description

Product name:
Wireless Headphones

If one replica shows the old description for a few seconds, the consequences may be minor.

Eventual consistency may be acceptable.

Available inventory

Suppose:

Available quantity = 1

Two users attempt to purchase the last item.

If two independent replicas both believe:

Available quantity = 1

both might accept the order.

Now we have oversold inventory.

This may require stronger coordination.

A useful way to think about consistency is:

How wrong is the system allowed to be,
and for how long?

That question is often more useful than asking:

Should we use strong or eventual consistency?

Different parts of the same application can use different consistency guarantees.

A single system might use:

Payments      -> Stronger consistency
Inventory     -> Stronger coordination
Search index  -> Eventual consistency
Analytics     -> Eventual consistency
Notifications -> Asynchronous/eventual delivery

Distributed system design is usually about choosing the correct guarantees for each operation.

5. Distributed Transactions

A traditional database transaction provides ACID properties.

For example:

BEGIN

Deduct $100 from Account A
Add $100 to Account B

COMMIT

Either everything succeeds:

A = A - 100
B = B + 100

Or everything is rolled back.

But what happens when the operations belong to different services?

Order Service
      |
      +----> Payment Service
      |
      +----> Inventory Service
      |
      +----> Shipping Service

Creating an order may involve:

1. Reserve inventory
2. Charge payment
3. Create shipment
4. Confirm order

Now there is no single local database transaction covering all services.

This creates the distributed transaction problem.

Two-Phase Commit

One traditional approach is Two-Phase Commit, or 2PC.

A coordinator asks participants:

Coordinator:
"Can you commit?"

Participants respond:

Payment: Yes
Inventory: Yes
Shipping: Yes

Then:

Coordinator:
"Commit."

Conceptually:

        Coordinator
       /     |      \
      v      v       v
Payment Inventory Shipping

      PREPARE
         |
         v
      COMMIT

2PC can provide strong transactional coordination.

But it introduces problems:

Increased latency
Tight coupling
Blocking
Coordinator failure complexity
Reduced availability

For many microservice systems, distributed database transactions are avoided.

Instead, systems use patterns such as:

Sagas
Compensating transactions
Transactional outbox
Idempotent operations
Reliable messaging

Saga Pattern

Instead of one large transaction:

Reserve Inventory
Charge Payment
Create Shipment

each service performs its own local transaction.

If a later operation fails, earlier operations are compensated.

For example:

1. Reserve inventory       ✓
2. Charge payment          ✓
3. Create shipment         X

The system may execute:

Refund payment
Release inventory

The flow becomes:

Reserve Inventory
        |
        v
Charge Payment
        |
        v
Create Shipment
        |
        X
        |
        v
Refund Payment
        |
        v
Release Inventory

This introduces an important reality:

Distributed transactions are often not about making failures disappear. They are about designing what should happen after partial success.

6. Network Failures

A network is not a reliable pipe between machines.

Requests can:

Fail to send
Arrive late
Be duplicated
Arrive out of order
Be dropped
Have their responses lost

Consider:

Client ---- Request ----> Service
Client <--- Response ---- Service

There are multiple failure points.

Client ---- X ----------> Service

The request may never arrive.

Or:

Client ---- Request ----> Service
Client <--- X ----------- Service

The service may successfully process the request, but the response may be lost.

This distinction matters enormously.

Suppose:

POST /payments

The server processes the payment.

Then the response disappears.

The client waits:

...
...
Timeout

Should the client retry?

If it retries blindly:

POST /payments
POST /payments

it might charge the customer twice.

This is why network failure cannot be solved by simply saying:

Retry if something fails.

Retries require idempotency, deduplication, and careful timeout policies.

7. Partial Failures

In a monolithic application, we often think about system failure as:

System is working
OR
System is down

Distributed systems are different.

One part can fail while everything else continues.

For example:

                API Gateway
                     |
        +------------+------------+
        |            |            |
        v            v            v
   User Service  Order Service  Payment Service
        ✓            ✓            X

The system is partially healthy.

What should happen?

The answer depends on the operation.

Maybe the system can:

Show products       ✓
View profile        ✓
Create order        ✓
Process payment     X

Or perhaps the order can be created in a pending state:

Order Created
Status = PAYMENT_PENDING

The payment can be retried later.

Partial failure forces us to design systems in terms of degradation.

Instead of asking:

Is the system up?

we often need to ask:

Which capabilities are currently available?

This leads to patterns such as:

Graceful degradation
Circuit breakers
Bulkheads
Fallback responses
Asynchronous processing

8. Timeouts

A timeout is one of the most important mechanisms in distributed systems.

Without timeouts, a system can wait forever.

Imagine:

Service A ---> Service B

Service B becomes extremely slow.

If Service A waits indefinitely:

Request 1 -> waiting
Request 2 -> waiting
Request 3 -> waiting
Request 4 -> waiting
...

Eventually:

Threads become exhausted
Connections become exhausted
Queues grow
Latency increases
More requests time out

One slow dependency can create a cascading failure.

Therefore:

Every remote call should have a timeout.

But choosing the timeout is not trivial.

Too short:

Request is actually healthy
        |
        v
Client gives up too early

Too long:

Dependency is failing
        |
        v
Resources remain occupied

Timeouts should usually reflect:

Expected latency
Tail latency
Request importance
Retry strategy
Overall request deadline

A useful concept is the deadline.

Suppose an API request has:

Total deadline = 2 seconds

You should not allow every internal dependency to wait 2 seconds independently.

Otherwise:

API
 |
 +-- Service A: 2s
 |
 +-- Service B: 2s
 |
 +-- Service C: 2s

The total latency may exceed the user's acceptable wait time.

Instead, time should be budgeted.

Request deadline: 2 seconds

Authentication: 200ms
Inventory:      500ms
Payment:        800ms
Remaining:      Processing

Timeouts are not merely error-handling settings.

They are part of system capacity and failure management.

9. Retries

Retries are necessary because many failures are temporary.

For example:

Request
   |
   X Network issue
   |
Retry
   |
   ✓ Success

But retries can also make outages dramatically worse.

Suppose a service is already overloaded.

10,000 requests

begin failing.

Every client retries three times.

Now:

Original traffic: 10,000
Retry traffic:    30,000

The overloaded service receives even more work.

This is called a retry storm.

A good retry strategy usually includes:

Limited retries

Do not retry forever.

Attempt 1
Attempt 2
Attempt 3
Stop

Exponential backoff

Instead of:

Retry immediately
Retry immediately
Retry immediately

use increasing delays:

1 second
2 seconds
4 seconds
8 seconds

Conceptually:

delay = baseDelay × 2^attempt

Jitter

If thousands of clients retry at exactly the same time:

Service fails
       |
       v
All clients retry after 1 second
       |
       v
Huge traffic spike

Jitter adds randomness:

1.2s
0.8s
1.5s
1.1s

This spreads retries over time.

The important rule is:

Retry only when repeating the operation is safe and likely to succeed.

That brings us to idempotency.

10. Idempotency

An operation is idempotent if performing it multiple times has the same effect as performing it once.

For example:

PUT /users/123/status

{
    "status": "ACTIVE"
}

Sending the request once:

status = ACTIVE

Sending it ten times:

status = ACTIVE

The final state is the same.

Compare that with:

POST /payments

{
    "amount": 100
}

Sending this request twice may produce:

Charge $100
Charge $100

Total:

$200

That is not safe to retry without additional protection.

Idempotency Keys

A common approach is an idempotency key.

The client sends:

POST /payments
Idempotency-Key: 550e8400

The server stores the key with the result.

Key: 550e8400
Result: Payment successful

If the client retries:

POST /payments
Idempotency-Key: 550e8400

the server recognizes that the operation has already been processed.

Instead of charging again:

Return previous result

Conceptually:

Request
   |
   v
Idempotency Key Seen?
   |
   +---- No ----> Process Operation
   |                   |
   |                   v
   |               Store Result
   |
   +---- Yes ---> Return Existing Result

This is extremely useful for:

Payments
Order creation
Message processing
Webhooks
Event consumers

Idempotency is one of the core tools for making retries safe.

11. Ordering

In a distributed system, messages do not automatically arrive in the same order in which they were sent.

Suppose:

Event 1: Order Created
Event 2: Order Paid

You might expect:

Consumer receives:

Order Created
Order Paid

But distributed delivery can produce:

Order Paid
Order Created

Or:

Order Created
Order Created
Order Paid

Or even:

Order Paid

while the first event is delayed.

Ordering guarantees are usually more complicated than people initially expect.

Global Ordering

A global order means every participant agrees on one universal sequence:

Providing this across a large distributed system requires significant coordination.

It can reduce scalability and availability.

Partition or Entity Ordering

Often we do not need global ordering.

We only need ordering for a particular entity.

For example:

Order 101:
Created
Paid
Shipped
Delivered

These events must be ordered.

But there may be no need to coordinate them with events for:

Order 202
Order 303
Order 404

A system can partition messages by:

orderId

Then:

Partition A -> Order 101 events
Partition B -> Order 202 events
Partition C -> Order 303 events

This is a common trade-off:

Preserve ordering where it matters instead of trying to order everything globally.

12. Deduplication

Distributed messaging systems frequently provide at-least-once delivery.

This means:

A message will be delivered one or more times.

The phrase "one or more times" is important.

A message can be duplicated.

Suppose:

Message: OrderCreated(orderId=101)

The consumer processes it:

Create invoice

But before acknowledging:

Consumer crashes

The message broker does not know whether processing completed.

So it sends the message again.

OrderCreated(orderId=101)

Now the consumer might create another invoice.

The solution is deduplication.

For example:

Processed Events

event-001 -> processed
event-002 -> processed
event-003 -> processed

When an event arrives:

Already processed?

If:

Yes -> Ignore or return previous result
No  -> Process and record

This is closely related to idempotency.

A useful design principle is:

Assume that messages can be delivered more than once unless the system explicitly provides and can truly preserve a stronger guarantee.

Even when infrastructure provides advanced guarantees, application-level idempotency is often valuable because failures can happen around the boundaries of message processing and side effects.

13. Backpressure

Imagine a producer generates work faster than a consumer can process it.

Producer
   |
   | 1000 events/sec
   v
Queue
   |
   | 100 events/sec
   v
Consumer

The queue grows:

Eventually:

Memory is exhausted
Disk fills
Latency becomes unacceptable
The consumer cannot catch up

Backpressure is a mechanism for telling the producer:

Slow down.

Conceptually:

Producer ---> Consumer

Consumer:
"I cannot process data at this rate."
        |
        v
Producer reduces rate

Backpressure can be implemented through:

Bounded queues
Flow control
Rate limiting
Consumer acknowledgments
Pull-based consumption
Reactive streams
Load shedding

The important principle is:

Infinite buffering is not a solution to overload.

A queue can absorb a temporary burst.

It cannot fix a system where:

Input rate > processing rate

forever.

If:

Producer = 1000 requests/sec
Consumer = 100 requests/sec

then the system must eventually:

Process faster
Scale consumers
Slow producers
Reject work
Drop non-critical work

Backpressure makes this limitation explicit instead of allowing uncontrolled resource exhaustion.

14. Load Balancing

When multiple instances provide the same service:

                 +--> Service A
Client --> LB ---+
                 +--> Service B
                 |
                 +--> Service C

the load balancer decides where requests go.

Common strategies include:

Round Robin

Request 1 -> A
Request 2 -> B
Request 3 -> C
Request 4 -> A

Simple and effective when instances have similar capacity.

Least Connections

Send the next request to the instance with the fewest active connections.

A: 100 connections
B: 20 connections
C: 50 connections

Next request -> B

This can be useful when request duration varies.

Weighted Load Balancing

Not every server has the same capacity.

For example:

Server A: weight 1
Server B: weight 2
Server C: weight 4

Server C receives more traffic.

Health Checks

Load balancing is not only about distributing traffic.

The load balancer also needs to know which instances should receive traffic.

Health Check

Service A -> Healthy
Service B -> Healthy
Service C -> Unhealthy

Then:

Requests -> A or B

A critical distinction is:

Process is running

does not necessarily mean:

Service is healthy

A process might be alive while:

Its database is unreachable
It is stuck
Its thread pool is exhausted
It is unable to process requests

Health checks should therefore be designed carefully.

15. Service Discovery

In a distributed system, services move.

Instances may:

Start
Stop
Restart
Scale up
Scale down
Move to different machines
Receive different IP addresses

Hardcoding addresses is not practical.

Imagine:

Payment Service
10.0.0.12

The machine fails.

A new instance starts:

Payment Service
10.0.0.57

Every client using the old address now fails.

Service discovery solves this problem.

Instead of saying:

Connect to 10.0.0.12

the application says:

Connect to payment-service

A discovery mechanism resolves:

payment-service

to healthy instances:

10.0.0.21
10.0.0.34
10.0.0.57

Conceptually:

              Service Registry

          payment-service
        /        |         \
       v         v          v
   Instance A Instance B Instance C

Client-Side Discovery

The client queries the registry.

Client
   |
   v
Service Registry
   |
   v
Available Instances
   |
   v
Client chooses instance

The client performs discovery and often load balancing.

Server-Side Discovery

The client sends requests to an intermediary:

Client
   |
   v
Load Balancer / Gateway
   |
   +----> Service A
   |
   +----> Service B
   |
   +----> Service C

The intermediary handles discovery and routing.

Modern platforms often provide service discovery through:

DNS
Container orchestration platforms
Service registries
Cloud infrastructure
Service meshes

The implementation varies.

The underlying problem remains the same:

How can one service reliably find another when the set of available machines constantly changes?

16. How These Concepts Work Together

The most important thing about distributed systems is that these concepts are not independent.

Consider a simple request:

Create Order

The request travels through:

Client
   |
   v
Load Balancer
   |
   v
Order Service
   |
   +----> Inventory Service
   |
   +----> Payment Service
   |
   +----> Message Broker

Now imagine this sequence:

1. Order Service reserves inventory
2. Payment Service processes payment
3. Payment response is lost
4. Order Service times out
5. Order Service retries payment
6. Payment receives duplicate request
7. Payment uses idempotency key
8. Payment returns existing result
9. Order Service publishes OrderPaid event
10. Event is delivered twice
11. Consumer deduplicates the event
12. Shipping Service processes the order

In one flow, we used:

Network failure handling
Timeouts
Retries
Idempotency
Deduplication
Asynchronous messaging

Now add a network partition.

Perhaps some replicas temporarily disagree about inventory.

Now consistency becomes important.

If traffic suddenly increases:

100 requests/sec
        |
        v
100,000 requests/sec

then:

Load balancing distributes traffic
Backpressure protects downstream systems
Timeouts prevent resource exhaustion
Retries must be controlled to avoid making overload worse

This is why distributed systems cannot be designed by choosing individual patterns in isolation.

Every decision affects other parts of the system.

17. A Better Mental Model for Distributed Systems

A common mistake is designing a distributed system as if it were a monolith split across multiple machines.

That approach leads to dangerous assumptions:

Remote call == local function call
Network == reliable
Request == delivered once
Response == operation result
Timeout == operation failed
Messages == ordered
Data == immediately visible everywhere
System failure == everything is down

None of these assumptions are universally true.

A better mental model is:

A distributed system consists of independent components
communicating through an unreliable network.

From that one idea, several design principles follow.

Assume partial failure

One dependency can fail while everything else works.

Design for degraded behavior.

Assume requests can be duplicated

Make important operations idempotent.

Assume responses can be lost

A timeout does not prove that an operation failed.

Assume messages can arrive late

Design workflows that can tolerate delayed processing.

Assume ordering is limited

Preserve ordering where necessary rather than globally.

Assume replicas can temporarily disagree

Choose consistency guarantees based on business requirements.

Assume overload will happen

Use backpressure, bounded queues, and load shedding.

Assume instances will move

Use service discovery instead of hardcoded addresses.

18. The Distributed Systems Checklist

When designing a new service or workflow, ask these questions.

Failure

What happens if this dependency is unavailable?

Timeout

How long should we wait?

Retry

Is this failure temporary?
How many times should we retry?

Idempotency

What happens if the same request is processed twice?

Consistency

Can this data be temporarily stale?

Ordering

Does processing order actually matter?
If yes, for which entity?

Deduplication

What happens if the same event arrives multiple times?

Backpressure

What happens if producers generate work faster than consumers can process it?

Partial failure

Can the system provide a reduced level of functionality?

Discovery

How do services find healthy instances?

Transactions

What happens when step 3 succeeds but step 4 fails?
How is the system compensated or recovered?

These questions often reveal design problems before the system reaches production.

Conclusion

The hardest part of distributed systems is not distributing the code.

It is dealing with uncertainty.

You cannot always know:

Whether a request arrived
Whether an operation completed
Whether another node is actually down
Whether a response was lost
Whether data is current
Whether a message will arrive once or multiple times
Whether another service is slow or permanently unavailable

Because of this, reliable distributed systems are built around explicit assumptions about failure.

CAP theorem helps us reason about consistency and availability during partitions.

Consistency models define what clients are allowed to observe.

Distributed transaction patterns help coordinate work across independent services.

Timeouts prevent systems from waiting forever.

Retries recover from temporary failures, but must be controlled.

Idempotency makes repeated operations safe.

Deduplication handles repeated messages.

Ordering ensures that operations are processed in the sequence that actually matters.

Backpressure prevents overload from becoming uncontrolled failure.

Load balancing distributes work across available capacity.

Service discovery allows services to find each other in an environment where machines constantly change.

The most important lesson is this:

In distributed systems, failure is not an exceptional case added after the happy path. Failure is part of the normal execution model.

Once you start designing with that assumption, many distributed systems concepts begin to connect.

A timeout leads to a retry.

A retry requires idempotency.

Idempotency helps handle duplicates.

Duplicates appear because delivery is uncertain.

Uncertain delivery exists because networks fail.

Networks fail independently, creating partial failures.

Partial failures force us to decide between availability and consistency.

And the entire system must still manage traffic, locate healthy services, control overload, and coordinate work across machines.

That is what makes distributed systems difficult.

And that is also what makes understanding their fundamentals essential for building reliable backend systems.

Distributed Systems Fundamentals: The Problems Every Backend Developer Must Understand

1. The Fundamental Problem: A Remote Call Is Not a Function Call

2. CAP Theorem

Consistency

Availability

Partition Tolerance

Choose consistency

Choose availability

3. Consistency Models

Strong Consistency

Eventual Consistency

Read-Your-Writes Consistency

4. Strong Consistency vs Eventual Consistency

Product description

Available inventory

5. Distributed Transactions

Two-Phase Commit

Saga Pattern

6. Network Failures

7. Partial Failures

8. Timeouts

9. Retries

Limited retries

Exponential backoff

Jitter

10. Idempotency

Idempotency Keys

11. Ordering

Global Ordering

Partition or Entity Ordering

12. Deduplication

13. Backpressure

14. Load Balancing

Round Robin

Least Connections

Weighted Load Balancing

Health Checks

15. Service Discovery

Client-Side Discovery

Server-Side Discovery

16. How These Concepts Work Together

17. A Better Mental Model for Distributed Systems

Assume partial failure

Assume requests can be duplicated

Assume responses can be lost

Assume messages can arrive late

Assume ordering is limited

Assume replicas can temporarily disagree

Assume overload will happen

Assume instances will move

18. The Distributed Systems Checklist

Failure

Timeout

Retry

Idempotency

Consistency

Ordering

Deduplication

Backpressure

Partial failure

Discovery

Transactions

Conclusion