Part 9: Observability - Logs, Metrics, Traces - Complete Guide

Part 9: Observability (3 Pillars)

Logs, Metrics, and Traces for Debugging Microservices - Complete Guide

📌 Quick Summary (30 seconds):
Three pillars of observability: Logs (detailed events, "what happened?"), Metrics (aggregated numeric data, "how much/how often?"), Traces (request flow across services, "where did it go?"). Together they provide complete visibility into system behavior. In microservices, observability is NOT optional — you cannot debug distributed systems without it.

❓ The Problem: Debugging Distributed Systems

Why Observability is CRITICAL in Microservices:

Request spans multiple services (hard to track)
Failure could be anywhere in the chain
No single log file to grep
Different teams own different services
Traditional debugging doesn't work

Without Observability:

"Payment failed" error message → Which service? Which instance? Why?
→ 50 services × 10 instances = 500 log files to search manually!
→ Hours wasted, customer unhappy

With Observability:

Trace ID links all services → Logs show the error → Metrics show pattern
→ Root cause found in minutes → Happy customer

📊 The Three Pillars of Observability

📝 LOGS
📄 Detailed, structured events
❓ "What happened?"
🛠️ ELK Stack, Loki, Splunk
📋 Example: "Payment failed for order 123: insufficient funds"

📊 METRICS
📈 Aggregated numeric data
❓ "How much? How often?"
🛠️ Prometheus, Grafana, Datadog
📋 Example: "Error rate = 15% in last 5 minutes"

🔍 TRACES
🗺️ Request flow across services
❓ "Where did it go?"
🛠️ Jaeger, Zipkin, OpenTelemetry
📋 Example: "Request: Gateway → Order → Payment → Inventory"

📝 Pillar 1: Logs (Deep Dive)

What are Logs?

Timestamped records of events
Structured (JSON) or unstructured (text)
Include context (request ID, user ID, etc.)
Every service writes to stdout/stderr
Collected by log aggregator (Fluentd, Logstash)

Log Levels (Importance)

ERROR - Something failed, needs attention
WARN - Something unexpected but working
INFO - Normal application events
DEBUG - Detailed troubleshooting info
TRACE - Very detailed (only during debugging)

// Good structured log (JSON) - Searchable, parseable
{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "ERROR",
  "service": "payment-service",
  "trace_id": "abc-123-def",
  "user_id": "user-456",
  "order_id": "order-789",
  "message": "Payment failed",
  "error": "insufficient_funds",
  "duration_ms": 145
}

// Bad log (unstructured) - Hard to search
"Payment failed at 10:30 for order 789 - insufficient funds"

📊 Pillar 2: Metrics (Deep Dive)

Types of Metrics

Counter - Only increases (request count, errors)
Gauge - Goes up and down (memory, connections)
Histogram - Distribution (latency percentiles)
Summary - Like histogram, client-side

Key Metrics to Monitor

Request Rate (RPS) - Traffic volume
Error Rate - Failure percentage
Latency (p50, p95, p99) - Response times
Resource Usage - CPU, Memory, Disk, Network
Business Metrics - Orders, Payments, Users

// Prometheus metrics example
# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="POST", endpoint="/payments", status="200"} 15234
http_requests_total{method="POST", endpoint="/payments", status="500"} 42

# HELP http_request_duration_seconds HTTP request latency
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.1"} 10000
http_request_duration_seconds_bucket{le="0.5"} 14500
http_request_duration_seconds_bucket{le="1.0"} 15000

🔍 Pillar 3: Distributed Tracing (Deep Dive)

Distributed Trace Example

Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736

┌─────────────────────────────────────────────────────────────────────┐
│ Service: API Gateway │
│ Span ID: 001 | Parent: null | Duration: 5ms │
└─────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Service: Order Service │
│ Span ID: 002 | Parent: 001 | Duration: 50ms │
└─────────────────────────────────────────────────────────────────────┘
│ │
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────────────────┐
│ Service: Payment Service │ │ Service: Inventory Service │
│ Span ID: 003 | Parent:002│ │ Span ID: 004 | Parent: 002 │
│ Duration: 200ms │ │ Duration: 30ms │
└─────────────────────────┘ └─────────────────────────────────────┘
Total Request Time = 5 + 50 + max(200, 30) = 255ms
Bottleneck identified: Payment Service (200ms)

// OpenTelemetry tracing example
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;

@Service
public class PaymentService {
    
    @Autowired
    private Tracer tracer;
    
    public PaymentResponse processPayment(PaymentRequest request) {
        // Start a new span
        Span span = tracer.spanBuilder("processPayment")
            .setAttribute("order.id", request.getOrderId())
            .setAttribute("amount", request.getAmount())
            .startSpan();
            
        try (Scope scope = span.makeCurrent()) {
            // Business logic
            PaymentResponse response = paymentProcessor.charge(request);
            span.setAttribute("payment.status", response.getStatus());
            return response;
        } catch (Exception e) {
            span.recordException(e);
            span.setStatus(StatusCode.ERROR, e.getMessage());
            throw e;
        } finally {
            span.end();
        }
    }
}

🔄 How Logs, Metrics, and Traces Work Together

Unified Observability Flow

User Request → Trace ID: abc-123
│
┌───────────────────┼───────────────────┐
│ │ │
▼ ▼ ▼
Gateway Order Service Payment Service
Span: 5ms Span: 50ms Span: 200ms
Log: "Received" Log: "Processing" Log: "FAILED - insufficient funds"
Metrics: RPS++ Metrics: RPS++ Metrics: ERROR++

Alert: Error rate > 10% at 10:30:05
↓
Query traces for that time period → Find trace abc-123
↓
View logs for trace abc-123 → "insufficient funds" error
↓
Root cause identified in 2 minutes!

📋 When to Use Each Pillar

Scenario	Primary Tool	Secondary	Why
"Why did this specific order fail?"	Logs	Traces	Logs contain detailed error message
"Is the system healthy right now?"	Metrics	Health Checks	Metrics show current error rate, latency
"Which service is slow?"	Traces	Metrics	Traces show latency per service
"How many users are active?"	Metrics	Logs	Request rate metric aggregates across all instances
"We need to scale up"	Metrics	None	CPU, memory, request rate show capacity needs
"What happened between 2-3 PM yesterday?"	Logs + Metrics	Traces	Historical data from both pillars

🛠️ Popular Observability Tools

.=志Commercial, developer-friendly .=志Future-proof instrumentation

Category	Tool	Key Features	Best For
Logs	ELK Stack (Elasticsearch, Logstash, Kibana)	Full-text search, dashboards, alerting	Large-scale logging
	Loki (Grafana Labs)	Tight Grafana integration, cheaper storage	Kubernetes environments
	Splunk	Enterprise features, machine learning	Enterprise, compliance-heavy
Metrics	Prometheus + Grafana	Pull model, powerful querying (PromQL)	Most popular, cloud-native
	Datadog	SaaS, APM integration, easy setup	Commercial, all-in-one
	New Relic	SaaS, user-friendly dashboards
Traces	Jaeger	OpenTracing compatible, UI for trace search	Uber open source, popular choice
	Zipkin	Lightweight, simple deployment	Smaller deployments
	OpenTelemetry	Unified standard, vendor-neutral

🔗 Correlation ID - The Glue That Binds Everything

What is Correlation ID? A unique identifier passed through all services in a request chain. It ties logs, metrics, and traces together.

Correlation ID Flow

Client: POST /orders (Header: X-Correlation-ID: abc-123)
│
▼
Gateway: Logs with correlation_id=abc-123
│
▼
Order Service: Logs with correlation_id=abc-123
│
▼
Payment Service: Logs with correlation_id=abc-123
│
▼
All logs for a single request can be found by searching "correlation_id=abc-123"!

// Spring Boot - MDC (Mapped Diagnostic Context) for correlation ID
@Configuration
public class CorrelationIdFilter implements Filter {
    
    @Override
    public void doFilter(ServletRequest request, ServletResponse response, 
                         FilterChain chain) {
        HttpServletRequest req = (HttpServletRequest) request;
        
        // Extract or generate correlation ID
        String correlationId = req.getHeader("X-Correlation-Id");
        if (correlationId == null || correlationId.isEmpty()) {
            correlationId = UUID.randomUUID().toString();
        }
        
        // Add to MDC - will appear in all logs
        MDC.put("correlation_id", correlationId);
        
        try {
            chain.doFilter(request, response);
        } finally {
            MDC.clear();
        }
    }
}

// In logback configuration
// <pattern>%d{ISO8601} [%thread] %-5level %logger{36} - %msg - %X{correlation_id}%n</pattern>

⚡ Pros & Cons of Observability

✅ BENEFITS

Fast root cause analysis (minutes vs hours)
Proactive alerting before customers notice
Capacity planning with real data
Debug distributed transactions easily
Identify performance bottlenecks
Historical analysis for compliance

❌ CHALLENGES

Storage costs (logs, traces add up)
Performance overhead (tracing adds latency)
Complexity to set up and maintain
Noise (too many logs/metrics)
Sampling decisions (not all traces)
Cardinality issues with metrics

✅ Observability Best Practices

📝 Logs Best Practices

Use structured logging (JSON format)
Always include correlation ID
Use appropriate log levels
Don't log sensitive data (PII, passwords)
Centralize logs (ELK, Loki, Splunk)

📊 Metrics Best Practices

Monitor RED method: Rate, Errors, Duration
Monitor USE method: Utilization, Saturation, Errors
Set alerts with appropriate thresholds
Use histograms for latency
Avoid high-cardinality labels (user_id)

🔍 Traces Best Practices

Propagate trace ID across all services
Use head-based sampling for production
Add business context as attributes
Instrument all HTTP/DB/messaging calls
Set meaningful span names

📊 RED Method vs USE Method

🔴 RED Method (Service Level)

Rate - Requests per second
Errors - Percentage of failed requests
Duration - Time to process requests

Best for: User-facing services

⚙️ USE Method (Resource Level)

Utilization - % time resource is busy
Saturation - Queue length, waiting work
Errors - Error events

Best for: Infrastructure (CPU, memory, disk)

💡 Interview Power Phrases

"Observability is not optional in microservices. Logs tell you what happened, metrics tell you how much, and traces tell you where the request went. You need all three."

"We use correlation IDs to link logs, metrics, and traces across all services. A single request ID lets us trace the entire journey from API gateway to database."

"Our monitoring uses the RED method: Rate, Errors, and Duration for every service. We set alerts for error rate spikes and latency increases beyond p99 thresholds."

"We use OpenTelemetry for vendor-neutral instrumentation and Prometheus + Grafana for metrics, Jaeger for tracing, and Loki for logs."

📚 Quick Memory Reference

🔑 Key Terms

Logs - Detailed event records
Metrics - Aggregated numeric data
Traces - Request flow across services
Correlation ID - Links everything together
RED Method - Rate, Errors, Duration
USE Method - Utilization, Saturation, Errors

🧠 Mental Models

Logs = Diary entries (what happened)
Metrics = Heartbeat monitor (how healthy)
Traces = GPS route map (where it went)
Correlation ID = DNA linking evidence
RED method = Service health dashboard

📌 One-Line Summary

"Three pillars: Logs (details), Metrics (aggregates), Traces (flow) — together they make distributed systems debugable."

✅ Key Takeaways (Remember These 5 Points)

Three pillars - Logs, Metrics, Traces (need all three)
Correlation ID - Links everything together across services
RED Method - Rate, Errors, Duration for service health
USE Method - Utilization, Saturation, Errors for infrastructure
Observability is NOT optional - Without it, microservices are un-debugable

🎯 Remember This:
Without Observability → "Payment failed" → Search 500 log files → Hours wasted
With Observability → Trace ID links it all → 2 minutes → Problem fixed