Latency Engineering

Comprehensive guidance for diagnosing and reducing latency across the full software/hardware stack.

Mental Model

Latency = time delay between a cause and its observed effect. It is a distribution, not a single number. Always think in percentiles (p50, p95, p99, p99.9). Tail latency dominates real-world user experience far more than averages suggest.

Two fundamental laws:

• Little's Law: Concurrency = Throughput × Latency. Use it to size systems and understand queue dynamics.
• Amdahl's Law: Speedup = 1 / ((1-P) + P/N). Use it to set realistic expectations on parallelization gains.

Decision Framework: Where to Optimize First

1. Measure → identify where time is actually spent
2. Data locality?     → see references/data-latency.md
3. Computation cost?  → see references/compute-latency.md
4. Can't reduce it?   → see references/hiding-latency.md

Key Latency Constants (back-of-envelope)

Common Sources of Latency (checklist)

• Geographic distance — physical speed-of-light limit; only solution is colocation
• Last-mile network — Wi-Fi adds 5–50 ms on top of backbone latency
• Algorithmic complexity — O(n²) workloads hidden inside hot paths
• Serialization/deserialization — often surprisingly expensive
• Memory management / GC — stop-the-world pauses introduce tail latency spikes
• OS overhead — virtual memory, interrupts, context switching
• Lock contention — mutex waits serialize concurrent requests
• NUMA effects — remote memory access costs 2–3× local on multi-socket systems
• TCP Nagle algorithm / delayed ACKs — batches small packets, adds ~40ms delay on interactive traffic; disable with TCP_NODELAY
• Tail-at-scale — with fanout N, even rare slow calls affect most users

Measuring Latency Correctly

• Always capture full distributions, not just averages. Use histograms or eCDF plots.
• Report p95, p99, p99.9 for SLAs and debugging.
• Understand coordinated omission — naive benchmarks under-measure tail latency.
• Validate measurements: compare minimum to theoretical lower bound from latency constants table.

Reference Files

Load the relevant reference file based on the category of the problem:

When the user's question spans multiple areas, load all relevant files.