Latency Optimization

Comprehensive guide to reducing end-to-end latency in distributed systems - from network to application to database layers.

When to Use This Skill

• Optimizing response times for user-facing applications

• Creating latency budgets for distributed systems

• Implementing geographic routing strategies

• Reducing database query latency

• Optimizing API response times

• Understanding and measuring latency components

Latency Fundamentals

Understanding Latency

Latency Components:

Total Latency = Network + Processing + Queue + Serialization

┌─────────────────────────────────────────────────────────────┐ │ Request Journey │ │ │ │ Client ──► DNS ──► TCP ──► TLS ──► Server ──► DB ──► Back │ │ │ │ Components: │ │ ├── DNS Resolution: 0-100ms (cached: 0ms) │ │ ├── TCP Handshake: 1 RTT (~10-200ms) │ │ ├── TLS Handshake: 1-2 RTT (~20-400ms) │ │ ├── Request Transfer: depends on size │ │ ├── Server Processing: application-specific │ │ ├── Database Query: 1-1000ms typical │ │ └── Response Transfer: depends on size │ └─────────────────────────────────────────────────────────────┘

Key Metrics:

• P50: Median latency (50th percentile)
• P95: 95th percentile (tail latency starts)
• P99: 99th percentile (important for SLOs)
• P99.9: Three nines (critical systems)

Latency Numbers Every Developer Should Know

Latency Reference (2024 estimates):

Operation Time ───────────────────────────────────────────────────── L1 cache reference 1 ns L2 cache reference 4 ns Branch mispredict 5 ns L3 cache reference 10 ns Mutex lock/unlock 25 ns Main memory reference 100 ns Compress 1KB with Snappy 2,000 ns (2 μs) SSD random read 16,000 ns (16 μs) Read 1 MB from memory 50,000 ns (50 μs) Read 1 MB from SSD 200,000 ns (200 μs) Round trip same datacenter 500,000 ns (500 μs) Read 1 MB from network (1Gbps) 10,000,000 ns (10 ms) HDD random read 10,000,000 ns (10 ms) Round trip US East to US West 40,000,000 ns (40 ms) Round trip US to Europe 80,000,000 ns (80 ms) Round trip US to Asia 150,000,000 ns (150 ms)

Key Insights:

• Memory is 100x faster than SSD
• Same-datacenter is 80x faster than cross-continent
• Caching at any level provides huge wins

Latency Budget

Latency Budget Example (200ms target):

┌─────────────────────────────────────────────────────────────┐ │ 200ms Total Budget │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────┬──────────┬──────────┬──────────┬──────────┐ │ │ │ Network │ Auth │ Service │ DB │ Response │ │ │ │ 50ms │ 20ms │ 50ms │ 60ms │ 20ms │ │ │ └──────────┴──────────┴──────────┴──────────┴──────────┘ │ │ │ │ Breakdown: │ │ ├── Network (client → edge → origin): 50ms │ │ ├── Authentication/Authorization: 20ms │ │ ├── Service Processing: 50ms │ │ ├── Database Queries: 60ms │ │ └── Response Serialization + Transfer: 20ms │ └─────────────────────────────────────────────────────────────┘

Budget Rules:

1. Allocate budgets based on criticality
2. Leave 10-20% headroom for variance
3. Monitor P99 against budget
4. Alert when consistently over budget
5. Renegotiate budgets as system evolves

Network Latency Optimization

Geographic Routing

Geographic Routing Strategies:

1. GeoDNS Routing User IP ──► DNS Resolver ──► Nearest Server IP

   Pros: Simple, works everywhere Cons: DNS caching, IP geolocation inaccuracy

2. Anycast Routing Same IP advertised from multiple locations BGP routes to nearest (network topology)

   Pros: Instant failover, no DNS delay Cons: Requires BGP expertise, stateful sessions tricky

3. Load Balancer Geo-routing Global LB ──► Regional LB ──► Servers

   Pros: Fine-grained control, health checking Cons: Adds latency hop, more complex

Selection Guide: ┌──────────────────┬─────────────────────────────────────┐ │ Use Case │ Recommended Approach │ ├──────────────────┼─────────────────────────────────────┤ │ Static content │ Anycast CDN │ │ API services │ GeoDNS + Regional deployments │ │ Real-time apps │ Anycast + Connection persistence │ │ Stateful apps │ GeoDNS with session affinity │ └──────────────────┴─────────────────────────────────────┘

Protocol Optimization

Protocol-Level Optimizations:

1. HTTP/2 Benefits ├── Multiplexing (no head-of-line blocking) ├── Header compression (HPACK) ├── Server push (preemptive responses) └── Single connection (reduced handshakes)

   Latency Impact: 20-50% improvement typical

2. HTTP/3 (QUIC) Benefits ├── 0-RTT connection resumption ├── No TCP head-of-line blocking ├── Built-in encryption └── Connection migration (IP changes)

   Latency Impact: 10-30% over HTTP/2

3. TLS Optimization ├── TLS 1.3 (1-RTT handshake) ├── Session resumption (0-RTT) ├── OCSP stapling (no CA roundtrip) └── Certificate chain optimization

   Latency Impact: 50-200ms saved per connection

4. TCP Optimization ├── TCP Fast Open (TFO) ├── Increased initial congestion window ├── BBR congestion control └── Keep-alive for connection reuse

Connection Optimization

Connection Strategies:

1. Connection Pooling ┌─────────────────────────────────────────┐ │ Connection Pool │ │ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │ │ │Conn1│ │Conn2│ │Conn3│ │Conn4│ │ │ └──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘ │ └─────┼──────┼──────┼──────┼────────────┘ │ │ │ │ Reuse connections, avoid handshake cost

2. Preconnect/Prefetch <link rel="preconnect" href="https://api.example.com"> <link rel="dns-prefetch" href="https://cdn.example.com">

   Triggers early connection establishment

3. Connection Coalescing (HTTP/2) Multiple domains → single connection (When sharing same IP and certificate)

Application Latency Optimization

Caching Strategies

Caching Layers:

┌─────────────────────────────────────────────────────────────┐ │ Caching Hierarchy │ │ │ │ Browser ──► CDN Edge ──► App Cache ──► DB Cache ──► DB │ │ 1ms 10ms 20ms 50ms 100ms │ │ │ │ Each layer should catch most requests before next layer │ └─────────────────────────────────────────────────────────────┘

Cache Type Selection: ┌──────────────────┬─────────────────┬────────────────────────┐ │ Data Type │ Cache Location │ TTL Strategy │ ├──────────────────┼─────────────────┼────────────────────────┤ │ Static assets │ CDN + Browser │ Long (1 year), hashed │ │ API responses │ CDN + App │ Short (seconds-mins) │ │ Session data │ App (Redis) │ Session duration │ │ DB query results │ App (local/dist)│ Varies by query │ │ Computed results │ App │ Based on input staleness│ └──────────────────┴─────────────────┴────────────────────────┘

Async Processing

Async Patterns for Latency:

1. Background Processing Request ──► Validate ──► Queue ──► Response (fast) │ └──► Worker (async processing)

   User sees fast response, heavy work happens later

2. Parallel Requests Sequential: A(100ms) → B(100ms) → C(100ms) = 300ms

   Parallel: A(100ms) ─┐ B(100ms) ─┼──► 100ms total C(100ms) ─┘

3. Speculative Execution Start likely-needed work before confirmed Cancel if not needed Risk: Wasted resources if prediction wrong

4. Read-Your-Writes with Async Write ──► Queue ──► Response + Local Cache Update │ User sees their write immediately Backend processes asynchronously

Serialization Optimization

Serialization Format Comparison:

Format Encode Decode Size Human Speed Speed (relative) Readable ───────────────────────────────────────────────────── JSON Fast Fast Large Yes MessagePack V.Fast V.Fast Small No Protocol Buf Fast V.Fast V.Small No FlatBuffers Zero-copy V.Fast Small No Avro Fast Fast Small Schema

Recommendations:

• Internal services: Protocol Buffers or MessagePack
• Public APIs: JSON (compatibility) or gRPC (performance)
• High-throughput: FlatBuffers (zero-copy)
• Schema evolution: Avro or Protocol Buffers

Optimization Tips:

1. Avoid serializing unnecessary fields
2. Use streaming for large payloads
3. Compress large responses (gzip/brotli)
4. Consider binary formats for internal traffic

Database Latency Optimization

Query Optimization

Database Latency Patterns:

1. Index Optimization ❌ Full table scan: O(n) - slow ✓ Index lookup: O(log n) - fast ✓ Covering index: No table lookup needed

   Monitor: Slow query logs, EXPLAIN plans

2. Query Patterns ❌ N+1 queries: 1 + N roundtrips ✓ Batch queries: 1 roundtrip ✓ JOINs (when appropriate): 1 roundtrip

   Example: ❌ for user in users: get_orders(user.id) # N queries ✓ get_orders_for_users(user_ids) # 1 query

3. Connection Management ├── Connection pooling (avoid connection overhead) ├── Prepared statements (avoid parsing overhead) └── Connection proximity (same region as app)

4. Read Replicas ┌─────────────────────────────────────────┐ │ Writes ──► Primary │ │ Reads ──► Read Replica (lower latency)│ └─────────────────────────────────────────┘

Database Proximity

Database Placement Strategies:

1. Co-located Database App and DB in same availability zone Latency: <1ms Best for: Primary workloads

2. Same-Region Replica Read replica in same region Latency: 1-5ms Best for: Read scaling

3. Cross-Region Replica Replica in user's region Latency: Local (~5ms) vs cross-region (~100ms) Best for: Global read-heavy apps

4. Globally Distributed Database spans regions (CockroachDB, Spanner) Write latency: Higher (consensus) Read latency: Local Best for: Global consistency requirements

Measurement and Monitoring

Latency Measurement

Measurement Points:

1. Client-Side (Real User Monitoring) └── Measures actual user experience └── Includes network variability └── Tools: Browser timing API, RUM services

2. Edge/CDN Metrics └── Time to first byte (TTFB) └── Cache hit ratio └── Origin fetch time

3. Server-Side (APM) └── Request processing time └── Downstream service calls └── Database query time └── Tools: OpenTelemetry, APM vendors

4. Synthetic Monitoring └── Consistent measurement conditions └── Multiple geographic locations └── Baseline for comparison

Distributed Tracing: ┌─────────────────────────────────────────────────────────────┐ │ Request ──► Gateway ──► Service A ──► Service B ──► DB │ │ │ │ │ │ │ │ │ └───────────┴────────────┴────────────┴───────────┘ │ │ Trace ID links all spans together │ │ Each span has start time + duration │ └─────────────────────────────────────────────────────────────┘

Latency SLOs

Setting Latency SLOs:

1. Define Meaningful Metrics

   • P50: Typical experience
   • P95: Most users' worst case
   • P99: Tail latency for critical paths

2. Set Realistic Targets P50: 50ms (snappy feel) P95: 200ms (acceptable) P99: 500ms (degraded but functional)

3. Error Budget Approach If target is P99 < 500ms with 99.9% SLO:

   • Budget: 0.1% of requests can exceed 500ms
   • ~43 minutes per month of violations allowed

4. Alert Thresholds ├── Warning: P99 > 400ms (80% of budget) ├── Critical: P99 > 500ms (at budget) └── Page: P99 > 600ms for 5 minutes (over budget)

Common Anti-Patterns

Latency Anti-Patterns:

1. "Chattiness" ❌ Many small requests instead of batched ✓ Batch requests, use GraphQL, aggregate APIs

2. "Synchronous Chains" ❌ A → B → C → D (sequential) ✓ Parallelize independent calls, use async

3. "Unbounded Queries" ❌ SELECT * without limits or pagination ✓ Always paginate, limit result sets

4. "Cache Miss Storms" ❌ Cache expires, all requests hit origin ✓ Staggered TTLs, request coalescing, warm cache

5. "Logging in Hot Path" ❌ Synchronous logging on every request ✓ Async logging, sampling for high volume

6. "Premature Serialization" ❌ Serialize before knowing if needed ✓ Lazy serialization, stream when possible

7. "Ignoring Tail Latency" ❌ Only monitoring averages ✓ Track P95, P99, P99.9 for user experience

Best Practices

Latency Optimization Best Practices:

1. Measure First □ Establish baseline measurements □ Identify bottlenecks before optimizing □ Use distributed tracing □ Monitor percentiles, not just averages

2. Optimize Strategically □ Start with biggest bottlenecks □ Apply latency budgets □ Consider cost vs benefit □ Test optimizations under load

3. Network Layer □ Deploy close to users (CDN, edge) □ Use modern protocols (HTTP/2, HTTP/3) □ Optimize TLS (1.3, session resumption) □ Connection pooling and keep-alive

4. Application Layer □ Cache aggressively and appropriately □ Parallelize independent operations □ Use async processing for non-critical work □ Optimize serialization formats

5. Data Layer □ Index frequently queried columns □ Use read replicas for read-heavy loads □ Connection pooling □ Query optimization (avoid N+1)

6. Continuous Improvement □ Regular latency reviews □ Load testing with latency assertions □ Automated regression detection □ User experience correlation

Related Skills

• caching-strategies

• Application-level caching patterns

• multi-region-deployment

• Geographic distribution

• cdn-architecture

• Edge caching and delivery

• distributed-tracing

• End-to-end latency visibility