Part 8: Idempotency - Safe Retries in Distributed Systems - Complete Guide

Part 8: Idempotency

Executing the same operation multiple times = Same result as executing once - Complete Guide

📌 Quick Summary (30 seconds):
Idempotency ensures that performing the same operation multiple times produces the same result as doing it once. It is ESSENTIAL for safe retries in distributed systems. Without idempotency, a single network timeout could cause duplicate orders, double payments, or inconsistent data. Implement using idempotency keys, database constraints, or state-based updates.

❓ The Problem: Why Idempotency Matters

⚠️ Without Idempotency, This Happens:

User clicks "Pay Now" → Network timeout → User clicks again
Result: Customer charged TWICE
Order confirmation sends twice → Duplicate orders
Inventory deducted twice → Negative stock
Retry logic causes more damage than the original failure

The Retry Problem

Client Server Database
│ │ │
│──── POST /payment ──────►│ │
│ │───── Process ───────────►│
│ │ │
│ (NETWORK TIMEOUT) │ │
│◄────── Error ─────────────│ │
│ │ │
│──── RETRY POST /payment ─►│ │
│ │───── Process AGAIN ─────►│
│ │ │
│ │ 💥 DUPLICATE!
│ │ 💸 Double charge!
WITH IDEMPOTENCY: Second request is detected and ignored.

📖 What is Idempotency?

Formal Definition: An operation is idempotent if performing it multiple times has the same effect as performing it once.

✅ Idempotent Operations

SET status = 'CANCELLED'
DELETE FROM cart WHERE user_id = 123
UPDATE balance = 1000
GET /orders/123 (read-only)
PUT /users/123 (full replace)

❌ Non-Idempotent Operations

ADD 100 to balance
INSERT INTO orders (auto-increment)
POST /payments (no idempotency key)
UPDATE balance = balance + 100

🔧 Implementation Strategies

Strategy 1: Idempotency Key (Most Common)

Idempotency Key Flow

Client Server
│ │
│ POST /payments │
│ Header: Idempotency-Key: abc-123 │
│ ─────────────────────────────────────────►│
│ │
│ ┌──────▼──────┐
│ │ Check Redis │
│ │ for key │
│ └──────┬──────┘
│ │
│ ┌────────────────┼────────────────┐
│ │ Key NOT found │ Key FOUND │
│ ▼ ▼ │
│ Process request Return cached │
│ Store key in response │
│ Redis/DB │
│ │ │ │
│◄──────────────────────────┼────────────────┼───────────────│
│ Response Cached Response │

// Client: Generate and send idempotency key
POST /api/payments HTTP/1.1
Host: api.example.com
Idempotency-Key: 4e8b5c2a-7f3d-4e1a-9b2c-8d6e4f2a1b3c
Content-Type: application/json

{
    "amount": 100.00,
    "currency": "USD",
    "card": "****-****-****-1234"
}

// Server: Store processed keys
@PostMapping("/api/payments")
public PaymentResponse processPayment(
    @RequestHeader("Idempotency-Key") String idempotencyKey,
    @RequestBody PaymentRequest request) {
    
    // Check if already processed
    if (redis.hasKey(idempotencyKey)) {
        return redis.get(idempotencyKey);  // Return cached response
    }
    
    // Process payment
    PaymentResponse response = paymentService.process(request);
    
    // Store response with TTL (e.g., 24 hours)
    redis.set(idempotencyKey, response, Duration.ofHours(24));
    
    return response;
}

Strategy 2: Database Unique Constraints

-- Create table with unique constraint on business key
CREATE TABLE payments (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    payment_id VARCHAR(36) UNIQUE NOT NULL,  -- Idempotency key
    order_id VARCHAR(36) NOT NULL,
    amount DECIMAL(10,2),
    status VARCHAR(20),
    created_at TIMESTAMP
);

-- Insert will fail if payment_id already exists
INSERT INTO payments (payment_id, order_id, amount, status) 
VALUES ('abc-123', 'order-456', 100.00, 'COMPLETED');

-- Second insert with same payment_id throws duplicate key error

Strategy 3: State-Based Updates (Optimistic Locking)

-- ✅ Idempotent - Same result every time
UPDATE orders SET status = 'CANCELLED' WHERE order_id = '123';

-- ❌ NOT Idempotent - Each execution changes state
UPDATE orders SET retry_count = retry_count + 1 WHERE order_id = '123';

-- ✅ Idempotent with version check
UPDATE orders 
SET status = 'CANCELLED', version = version + 1 
WHERE order_id = '123' AND version = 5;

-- Version mismatch means already updated → ignore

Strategy 4: Message Deduplication

Kafka Message Deduplication

Kafka Topic Consumer Deduplication Store
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Message ID: │ │ │ │ │
│ msg-001 │───────────────►│ Consumer │─────────────►│ Check if │
└─────────────┘ │ │ │ ID exists │
└─────────────┘ └──────┬──────┘
│
┌─────────────────────┼─────────────────────┐
│ ID NOT FOUND │ ID FOUND │
▼ ▼ │
Process message Skip message │
Store ID in (duplicate) │
dedup store │

📋 HTTP Methods & Idempotency

HTTP Method	Idempotent?	Why?	Example
GET	✅ YES	Read-only, doesn't change state	GET /orders/123 → same response each time
PUT	✅ YES	Full replace, same data = same result	PUT /users/123 {"name":"John"} → always sets to John
DELETE	✅ YES	Deleting already deleted resource does nothing	DELETE /orders/123 → after first, resource gone
POST	❌ NOT by default	Creates new resource each time	POST /orders → creates NEW order each call
PATCH	⚠️ MAYBE	Depends on the operation (set vs increment)	PATCH with SET is idempotent, with ADD is not

⚠️ Important: POST is NOT idempotent by default. Always add idempotency keys to POST operations that can be retried (payments, orders, etc.).

📍 Where Idempotency is CRITICAL

System/Operation	Risk Without Idempotency	Solution
Payment Processing	Double charging customer	Idempotency key per transaction
Order Creation	Duplicate orders	Order ID + idempotency key
Inventory Updates	Negative stock / double deduction	Reservation ID + unique constraint
Event Processing	Duplicate events processed	Message ID deduplication
Email/SMS Sending	Customer gets duplicate messages	Request ID + dedup window
Wallet/Balance Updates	Money added twice	Version-based optimistic locking

⚡ Pros & Cons of Idempotency

✅ ADVANTAGES

Safe retries without side effects
Resilient to network failures
Prevents duplicate data (payments, orders)
Simplifies client retry logic
Essential for distributed transactions (Saga)
Enables at-least-once delivery guarantees

❌ CHALLENGES

Storage overhead for idempotency keys
Need to decide TTL for key storage
Additional complexity in API design
Distributed storage for keys (Redis cluster needed)
Keys can be guessed/abused if not random
Cleanup of old keys required

✅ Idempotency Key Best Practices

🔑 Key Generation (Client side)

Use UUID v4 (random, unique, non-sequential)
Never reuse same key for different operations
Same key = same operation, same parameters
Generate on client, not server
Store key in request metadata (header)

💾 Storage & TTL

Use Redis (fast, distributed, TTL support)
Set TTL = maximum retry window (24-48 hours)
Store response to return on duplicate
Clean up old keys automatically
Consider using database for critical operations

📋 Response Headers

Return same response for duplicate requests
Add Header: X-Idempotent-Response: true
Return HTTP 200 (not 201) for duplicate
Consider returning original timestamp

✈️ Real-World Example: Payment Processing

Complete Idempotent Payment Flow

Step 1: Client generates UUID: "pay_abc123"
Step 2: Send POST /payments with Idempotency-Key: pay_abc123
Step 3: Server checks Redis for key "pay_abc123"
Step 4: Key not found → Process payment
Step 5: Store {"status": "SUCCESS", "transaction_id": "txn_456"} in Redis with TTL 24h
Step 6: Return response to client
Step 7: Network timeout → Client retries with same key
Step 8: Server finds key → Returns cached response
Step 9: ✅ Customer charged ONCE, order created ONCE

// Complete Idempotency Key Implementation Example
@RestController
public class PaymentController {
    
    @Autowired
    private IdempotencyService idempotencyService;
    
    @Autowired
    private PaymentService paymentService;
    
    @PostMapping("/api/payments")
    public ResponseEntity<PaymentResponse> processPayment(
            @RequestHeader("Idempotency-Key") String idempotencyKey,
            @RequestBody PaymentRequest request) {
        
        // Check if already processed
        PaymentResponse cached = idempotencyService.get(idempotencyKey);
        if (cached != null) {
            return ResponseEntity.ok()
                .header("X-Idempotent-Response", "true")
                .body(cached);
        }
        
        // Process payment
        PaymentResponse response = paymentService.process(request);
        
        // Store for future retries (24 hours TTL)
        idempotencyService.store(idempotencyKey, response, Duration.ofHours(24));
        
        return ResponseEntity.ok(response);
    }
}

⚠️ Common Mistakes & How to Avoid Them

Mistake	Why It's Wrong	Correct Approach
Using timestamp as idempotency key	Not unique (multiple requests same ms)	Use UUID v4 or cryptographically random string
Storing keys forever	Infinite storage growth	Set TTL (e.g., 24-48 hours)
Reusing same key for different operations	Second request would be incorrectly ignored	Generate new key for each unique operation
Client generates key after network issues	Server can't detect duplicate if key changes	Client generates key BEFORE sending request
Not handling idempotency in async operations	Same webhook could be delivered twice	Use message ID deduplication

🔄 Idempotency in Saga Pattern

Why Idempotency is CRITICAL in Sagas:

Compensating transactions may be retried
Network failures can cause duplicate compensation calls
Without idempotency, refund could be processed twice
All forward AND compensating operations MUST be idempotent

Saga + Idempotency Example

Order Saga Flow:
1. Create Order → idempotency key: order_id
2. Process Payment → idempotency key: payment_id
3. Reserve Stock → idempotency key: reservation_id

If Step 3 fails, compensation triggers:
- Refund Payment: idempotency key: refund_{payment_id}
- Cancel Order: idempotency key: cancel_{order_id}
Same key used for compensation ensures no double refund!

💡 Interview Power Phrases

"Idempotency is essential for safe retries in distributed systems. We use idempotency keys to ensure that even if a client retries a request, we only process it once."

"Every POST operation that modifies state must be idempotent. We use UUID v4 as idempotency keys stored in Redis with 24-hour TTL."

"Without idempotency, retry logic causes more damage than the original failure. A single network timeout could double-charge a customer."

"In Saga pattern, both forward operations and compensating transactions must be idempotent. Otherwise, a retried compensation could refund the customer twice."

📚 Quick Memory Reference

🔑 Key Terms

Idempotency - Same operation multiple times = same result
Idempotency Key - Unique identifier for request
Idempotency Store - Cache of processed keys (Redis)
TTL - Time To Live for key storage
Safe Retry - Retry without side effects

🧠 Mental Models

Idempotent = Light switch (ON→ON→ON = same state)
Non-idempotent = Adding coins (each action changes state)
Idempotency Key = Unique receipt ID for each operation
Idempotency Store = "Already processed" ledger

📌 One-Line Summary

"Idempotency ensures retrying an operation doesn't cause duplicates - essential for payments, orders, and any system that retries failed requests."

✅ Key Takeaways (Remember These 5 Points)

Idempotent = Safe to retry - Same result whether executed once or 100 times
Idempotency Key - Most common implementation (UUID + Redis)
HTTP Methods - GET/PUT/DELETE are idempotent, POST is NOT by default
Critical for payments, orders, inventory, sagas - Without it, retries cause duplicates
Store with TTL - Keep keys for maximum retry window (24-48 hours)

🎯 Remember This:
Without Idempotency → Retry = Double charge, duplicate order, negative stock
With Idempotency → Retry = Safe, no side effects, happy customer