🛡️ Resilience Patterns
The Complete Resilience Stack — Timeout → Retry → Circuit Breaker → Fallback
🧠 Mental Model: The Restaurant Analogy
⏱️ TIMEOUT = "Waiting 10 minutes then giving up"
🔄 RETRY = "Calling again after 2 minutes"
🔌 CIRCUIT BREAKER = "Stop calling after 3 failures"
🔄 FALLBACK = "Order from another restaurant"
🎯 Think of calling a busy restaurant for delivery — these 4 patterns work exactly the same way!
📊 The Resilience Stack (Order of Protection)
1
⏱️ TIMEOUT
"Don't wait forever" — Limits max wait time
📞 Hang up after 30 seconds
⬇
2
🔄 RETRY
"Try again if temporary" — Exponential backoff
📞 Redial: 2s → 4s → 8s wait
⬇
3
🔌 CIRCUIT BREAKER
"Stop calling failing service" — Trip after 5 failures
⚡ Electrical breaker trips
⬇
4
🔄 FALLBACK
"Provide alternative response" — Graceful degradation
🍕 Order from backup restaurant
⏱️ TIMEOUT
connectTimeout: 3s
readTimeout: 5s
readTimeout: 5s
What: Limits max wait time for a call
Why: Prevents thread exhaustion
Rule: EVERY remote call needs timeout!
🔄 RETRY
100ms → 200ms → 400ms → 800ms
+ Jitter
+ Jitter
What: Re-attempt failed calls
When: Transient failures (network, 5xx)
Key: Exponential backoff + jitter
🔌 CIRCUIT BREAKER
CLOSED → OPEN → HALF-OPEN
What: Stops calling failing services
When: 5 failures in 10 seconds
Key: Prevents cascading failures
🔄 FALLBACK
return cached data
return default response
return default response
What: Alternative response on failure
When: After circuit breaker opens
Key: Graceful degradation
📋 Complete Payment Flow With Resilience
User clicks "Pay Now"
│
▼
╔═══════════════════════════════════════╗
║ 1️⃣ TIMEOUT (2 seconds max wait) ║
╚═══════════════════════════════════════╝
│ (Timeout occurs)
▼
╔═══════════════════════════════════════╗
║ 2️⃣ RETRY (Exponential backoff) ║
║ 100ms → 200ms → 400ms → 800ms ║
╚═══════════════════════════════════════╝
│ (All retries fail)
▼
╔═══════════════════════════════════════╗
║ 3️⃣ CIRCUIT BREAKER ║
║ CLOSED → OPEN (30 seconds) ║
╚═══════════════════════════════════════╝
│
▼
╔═══════════════════════════════════════╗
║ 4️⃣ FALLBACK ║
║ Return "Payment Pending" ║
║ Add to retry queue ║
╚═══════════════════════════════════════╝
│
▼
✅ User sees: "Your payment is being processed. We'll notify you."
(NO ERROR MESSAGE!)
User clicks "Pay Now"
│
▼
╔═══════════════════════════════════════╗
║ 1️⃣ TIMEOUT (2 seconds max wait) ║
╚═══════════════════════════════════════╝
│ (Timeout occurs)
▼
╔═══════════════════════════════════════╗
║ 2️⃣ RETRY (Exponential backoff) ║
║ 100ms → 200ms → 400ms → 800ms ║
╚═══════════════════════════════════════╝
│ (All retries fail)
▼
╔═══════════════════════════════════════╗
║ 3️⃣ CIRCUIT BREAKER ║
║ CLOSED → OPEN (30 seconds) ║
╚═══════════════════════════════════════╝
│
▼
╔═══════════════════════════════════════╗
║ 4️⃣ FALLBACK ║
║ Return "Payment Pending" ║
║ Add to retry queue ║
╚═══════════════════════════════════════╝
│
▼
✅ User sees: "Your payment is being processed. We'll notify you."
(NO ERROR MESSAGE!)
📊 Quick Comparison Table
| Pattern | What it does | When it triggers | Real-World Analogy |
|---|---|---|---|
| ⏱️ Timeout | Limits wait time | Call takes too long | Hanging up after 30 seconds |
| 🔄 Retry | Re-attempts failed call | Temporary failure | Redialing a busy number |
| 🔌 Circuit Breaker | Stops calling failing service | Repeated failures (5+ in 10s) | Tripped electrical breaker |
| 🔄 Fallback | Provides alternative response | All else fails | Ordering from backup restaurant |
🎯 Golden Rules of Resilience
1️⃣
Every remote call needs TIMEOUT
2️⃣
Use EXPONENTIAL BACKOFF with jitter for retries
3️⃣
CIRCUIT BREAKER prevents cascading failures
4️⃣
FALLBACK provides graceful degradation
"Timeout → Retry → Circuit Breaker → Fallback — Apply in this exact order!"
🎯 One-Line Summary
"The resilience stack is TIMEOUT first, then RETRY, then CIRCUIT BREAKER, then FALLBACK — apply in this exact order!"
"The resilience stack is TIMEOUT first, then RETRY, then CIRCUIT BREAKER, then FALLBACK — apply in this exact order!"