Part 6: Saga Pattern - Complete Guide to Distributed Transactions

Part 6: Saga Pattern

Managing Distributed Transactions Across Microservices - Complete Guide

📌 Quick Summary (30 seconds):
Saga pattern manages distributed transactions across microservices using a sequence of local transactions, each with a compensating action. Two implementations: Choreography (decentralized, event-driven) and Orchestration (centralized coordinator). Choose choreography for simplicity, orchestration for complex workflows with visibility needs.

❓ The Problem: No Distributed Transactions in Microservices

Why can't we use traditional ACID transactions?

No shared database between services
Distributed transactions (2PC/XA) don't scale well
Locks would be held across services (performance killer)
Network failures make coordination impossible (CAP theorem)

🔄 What is a Saga?

Saga = Sequence of Local Transactions + Compensations

Step 1: Order Service ──► Create Order (PENDING)
│
▼
Step 2: Payment Service ──► Process Payment
│
▼
Step 3: Inventory Service ──► Reserve Stock
│
▼
Step 4: Shipping Service ──► Create Shipment
│
▼
✅ SUCCESS - Order Complete

IF ANY STEP FAILS:

Step 3 fails (Out of Stock)
│
▼
Compensation 2: Payment Service ──► Refund Payment
│
▼
Compensation 1: Order Service ──► Cancel Order
│
▼
❌ Order Cancelled - Rollback Complete

📊 Saga vs ACID Transaction

Feature	ACID Transaction	Saga Pattern
Database Scope	Single Database	Multiple Services
Consistency Model	Strong (ACID)	Eventual (BASE)
Locking	Yes (holds locks)	No (no locks)
Isolation	Serializable	No isolation (dirty reads possible)
Scalability	Limited	High
Failure Handling	Automatic Rollback	Compensating Transactions
Complexity	Low	High
Best For	Single service operations	Cross-service workflows

🎭 1. Choreography (Event-Driven Saga)

Choreography Flow - No Central Controller

┌─────────────┐ ┌─────────────┐
│ Order │ ──event: OrderCreated──► │ Payment │
│ Service │ │ Service │
└─────────────┘ └──────┬──────┘
│ event: PaymentSuccess
▼
┌─────────────┐ ┌─────────────┐
│ Shipping │ ◄──event: StockReserved── │ Inventory │
│ Service │ │ Service │
└─────────────┘ └─────────────┘

✅ PROS

No central point of failure
Simple to implement for linear flows
Good scalability (decentralized)
Loose coupling between services
Easy to add new participants

❌ CONS

Hard to debug (no central view)
Risk of cyclic dependencies
Complex error handling
No central audit trail
Hard to manage complex workflows

🎼 2. Orchestration (Central Coordinator Saga)

Orchestration Flow - Central Coordinator

┌─────────────────────────────┐
│ SAGA ORCHESTRATOR │
│ (Central Coordinator) │
└──────────────┬──────────────┘
│
┌───────────────┬──────────────┼───────────────┬───────────────┐
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐
│ Order │ │ Payment │ │ Inventory │ │ Shipping │ │ User │
│ Service │ │ Service │ │ Service │ │ Service │ │ Service │
└───────────┘ └───────────┘ └───────────┘ └───────────┘ └───────────┘
Orchestrator tracks state and knows which step to execute next. If a step fails, Orchestrator triggers compensations in reverse order.

✅ PROS

Central visibility and control
Easier to debug (single place to check)
Clear audit trail
Simpler error handling
Supports complex branching logic

❌ CONS

Orchestrator is a single point of failure
Can become performance bottleneck
More infrastructure complexity
Orchestrator business logic can grow complex
Centralized = potential coupling

📋 Choreography vs Orchestration - Quick Comparison

Aspect	Choreography	Orchestration
Control	Decentralized	Centralized
Communication	Events	Commands (sync/async)
Visibility	Low (hard to trace)	High (central logs)
Debugging	Difficult	Easier
Scalability	Excellent	Limited by orchestrator
Single Point of Failure	No	Yes (orchestrator)
Complex Workflows	Hard to manage	Well suited
Cyclic Dependencies Risk	High	Low
Best For	Simple, linear workflows	Complex, branching workflows

🔄 Compensating Transactions Reference

⚠️ Golden Rule: Every step in a saga MUST have a compensating action defined.

Forward Transaction	Compensating Action	Idempotent?	Example
Create Order	Cancel Order	Yes (orderId)	Set status to CANCELLED
Process Payment	Refund Payment	Yes (paymentId)	Reverse charge on card
Reserve Stock	Release Stock	Yes (reservationId)	Add stock back to inventory
Book Hotel	Cancel Booking	Yes (bookingId)	Free up the room
Send Email	Send Reversal Email	Yes (emailId)	Send cancellation notice
Book Flight	Cancel Flight	Yes (PNR)	Refund or credit voucher
Update Cache	Revert/Invalidate Cache	Yes (cacheKey)	Remove or restore old value

🎯 When to Use Choreography vs Orchestration

🎭 Choose Choreography When:

Workflow has 2-4 steps (simple and linear)
Teams want full autonomy
You have good event-driven infrastructure (Kafka, RabbitMQ)
No need for central audit trail
Services can work independently
Low coordination overhead needed

📌 Example: Order → Payment → Inventory (linear)

🎼 Choose Orchestration When:

Workflow has complex branching/conditional logic
Central visibility and audit trail required
Multiple sagas share common steps
Need ability to resume sagas after failures
Compliance requires transaction logs
Business rules change frequently

📌 Example: Travel booking with flights, hotels, cars, insurance (branching)

✈️ Real-World Examples

Industry	Example Saga	Steps	Compensations
E-Commerce	Order Processing	Order → Payment → Inventory → Shipping	Cancel Order → Refund → Release Stock → Cancel Shipment
Travel Booking	Trip Reservation	Flight → Hotel → Car → Insurance	Cancel Flight → Cancel Hotel → Cancel Car → Cancel Insurance
Banking	Fund Transfer	Debit Account A → Credit Account B	Credit Account A → Debit Account B
Insurance	Claim Processing	Validate → Approve → Pay → Notify	Reverse Validation → Reject → Recover Payment → Recall Notification

⚠️ Common Pitfalls and Solutions

Pitfall	Symptom	Solution
Non-idempotent operations	Duplicate payments on retry	Use idempotency keys + unique database constraints
Missing compensations	Partial rollback, data inconsistency	Design compensation for EVERY forward step
Long-running sagas	Resource locks, degraded performance	Add timeouts, checkpoints, resume capability
No visibility	Cannot debug failures	Central saga state repository + monitoring dashboard
Compensation failure	Stuck in inconsistent state	Dead letter queue + manual intervention alerts
No timeout handling	Saga never completes	Implement saga timeouts and scheduled compensation
Shared database between services	Services not independent	Enforce database per service

🔑 Idempotency in Sagas (Critical!)

Why Idempotency Matters: The same compensation may be called multiple times due to network retries.

Idempotent vs Non-Idempotent Operations

✅ IDEMPOTENT (Safe to retry):
• SET balance = 1000 → Same result every time
• UPDATE status = 'CANCELLED' WHERE id = 123
• DELETE FROM cart WHERE user_id = 456

❌ NON-IDEMPOTENT (NOT safe to retry):
• ADD 100 to balance → Each retry adds another 100
• INSERT INTO orders → Each retry creates duplicate
• UPDATE balance = balance + 100 → Same problem

✅ Saga Best Practices

Design & Implementation

Design compensations FIRST, then forward transactions
Make ALL operations idempotent (use idempotency keys)
Store saga state persistently (database, not just memory)
Implement timeouts for every step
Monitor sagas actively (detect stuck/long-running)

Testing & Operations

Test compensation paths as thoroughly as success paths
Keep sagas short (fewer steps = lower failure probability)
Log every step with correlation ID
Implement dead letter queue for failed compensations
Have manual intervention plan for unrecoverable states

🛠️ Popular Saga Frameworks

Framework	Type	Language	Key Feature
Axon Framework	Orchestration	Java	Event Sourcing + CQRS built-in
Camunda	Orchestration (BPMN)	Java	Visual workflow designer
Temporal.io	Orchestration	Java/Go	Durable execution, fault tolerance
Apache Kafka	Choreography	Any	Event-driven sagas
Spring Cloud	Both	Java	Spring ecosystem integration
MassTransit	Both	.NET	.NET ecosystem

💡 Interview Power Phrases

"Saga replaces ACID with eventual consistency using compensating transactions. Each step has a corresponding compensation that undoes its effect if later steps fail."

"We use choreography for simple, linear workflows where services can work independently. We use orchestration for complex workflows where central visibility and audit trail are required."

"The hardest part of sagas isn't the forward path - it's the compensation. If you can't design a safe compensating transaction, you shouldn't split that operation into a separate microservice."

"All operations in a saga must be idempotent. We use idempotency keys to ensure that retries don't cause duplicate effects."

📚 Quick Memory Reference

🔑 Key Terms

Saga - Sequence of local transactions
Compensation - Undo operation for each step
Choreography - Decentralized, event-driven
Orchestration - Centralized coordinator
Idempotency - Safe retries
Eventually Consistent - BASE not ACID

🧠 Mental Models

ACID = Single chef in one kitchen
Saga = Multiple chefs coordinating via messages
Choreography = Dancers without a leader
Orchestration = Conductor leading the orchestra
Compensation = Undo button for each action

📌 One-Line Summary

"Saga replaces distributed transactions with compensations - trade ACID for BASE, locks for events, strong consistency for eventual consistency."

✅ Key Takeaways (Remember These 5 Points)

Saga ≠ ACID - No locks, eventual consistency, compensations instead of rollback
Two patterns: Choreography (decentralized events) vs Orchestration (central coordinator)
Every step needs a compensation - Design forward and backward together
Idempotency is mandatory - Same operation multiple times = same result
Store saga state - For recovery, monitoring, and debugging

🎯 Remember This:
Choreography = Simple, scalable, hard to debug
Orchestration = Complex, visible, centralized control
Both = Compensations + Idempotency + Eventual Consistency