Latency Principles

Based on "Latency: Reduce delay in software systems" by Pekka Enberg (Manning, 2026).

This skill provides a systematic framework for minimizing delay in software systems, covering the entire stack from Physics and Hardware to Application Architecture and User Experience.

Foundational Concepts

• What is Latency?: The time delay between a cause and its observed effect.
• Latency vs. Bandwidth: You can always add more bandwidth (more links), but you are stuck with bad latency unless you optimize the path.
• Impact on User Experience (UX): < 100ms (immediate), < 1s (instant), > 10s (slow).
• Measuring Correctly: Use percentiles (p95, p99) to capture tail latency. Avoid averages. Avoid Coordinated Omission by using fixed-interval benchmarking.

Modeling Performance

Use these laws to establish theoretical bounds and size systems:

• Little's Law: Concurrency (C) = Throughput (T) * Latency (L)
  • Usage: Calculate required concurrency to support a target throughput at a given latency.
  • Example: If p99 latency is 50ms and you need 1000 RPS, you need 1000 * 0.05 = 50 concurrent execution units.
• Amdahl's Law: Speedup = 1 / ((1 - P) + (P / N))
  • P: Portion of program that is parallelizable.
  • N: Number of processors.
  • Usage: Understand the limits of parallelization. If 50% of your code is serial, the max speedup is 2x, regardless of how many cores you add.

Measurement & Visualization

Visualizing latency distributions is critical for identifying tail behavior:

1. Histograms: Show frequency of samples. Good for seeing the "mode" (most common latency) and the spread.
2. HDR Histograms: Plot latency against percentiles (p50, p99, etc.) on a log scale. Essential for p99.9+ analysis.
3. eCDF (Empirical Cumulative Distribution Function): A smooth curve showing the probability that a request completes within a given time. Directly answers SLA compliance questions.

Tooling:

• Use scripts/ping_collector.py to gather data without Coordinated Omission.
• Use scripts/visualize_latency.py to generate Histogram, HDR, and eCDF plots.

Quick Decision Guide

Part 1: Fundamentals (Start Here)

Core theory and diagnostic approaches.

• references/principles.md: Definitions of Little's Law, Amdahl's Law, Tail Latency, Compounding Latency, and Latency Distribution.
• references/diagnostic_checklist.md: Step-by-step checklist for identifying bottlenecks across Hardware, OS, and Runtime.

Part 2: Data Layer (Access Optimization)

Optimizing data access is often the highest-leverage activity for reducing latency.

• Colocation (Pattern: Move Compute to Data):
  • Edge Computing: Use Near Edge (points of presence) or Far Edge (on-device/IoT) to eliminate geographical distance.
  • Intranode: Colocate protocol handlers with application threads. Turn off Nagle's Algorithm (TCP_NODELAY) to prevent packet batching delays.
  • Kernel-bypass: Use techniques like DPDK to eliminate OS stack overhead.
• Replication (Pattern: Trade Consistency for Latency):
  • Leaderless/Multi-Leader: Allows local writes to reduce write latency, at the cost of complex conflict resolution.
  • Consistency Models: Choose Eventual Consistency or Read-your-writes to avoid synchronous coordination (Strong Consistency) overhead.
• Partitioning (Pattern: Divide and Conquer):
  • Horizontal Sharding: Splitting data to increase parallel throughput.
  • Request Routing: Use Direct Routing (client-side) to avoid the extra hop of a proxy.
  • Mitigate Hot Partitions: Use over-partitioning to balance skewed workloads.
• Caching (Pattern: Memory is Faster than Disk):
  • Write-Behind Caching: Asynchronous writes to the data store to hide write latency.
  • Policy Selection: Use SIEVE or LRU for eviction.
  • Materialized Views: Precompute complex queries to eliminate runtime processing work.

See references/data_patterns.md for detailed implementation strategies.

Part 3: Compute Layer (Logic Acceleration)

Optimizing processing logic and synchronization to eliminate overhead.

• Eliminating Work (Pattern: The Fastest Code is Code that Doesn't Run):
  • Algorithmic Complexity: Replace O(n) scans with O(log n) trees or O(1) hash maps.
  • Zero-Copy Serialization: Use FlatBuffers instead of JSON to eliminate the parsing/unpacking step.
  • Memory Tuning: Avoid dynamic allocation (malloc/new) in hot paths. Use Object Pooling or Stack Allocation to prevent GC pauses or allocator lock contention.
  • Precomputation: Move work from runtime to build-time or startup.
• Wait-Free Sync (Pattern: Avoid Context Switches):
  • Mutual Exclusion Problems: Locks (Mutexes) cause expensive OS context switches (~µs).
  • Atomics: Use hardware primitives (CAS, fetch_add) for lock-free state updates.
  • Wait-Free Structures: Implement Ring Buffers (SPSC) using memory barriers to allow threads to communicate without ever blocking.
• Exploiting Concurrency (Pattern: Use Every Core):
  • Thread-per-core: Pin threads to physical cores to maximize cache locality and eliminate scheduler overhead.
  • Concurrency Models: Use Coroutines/Fibers for lightweight userspace multitasking or Actor Model for shared-nothing message passing.
  • SIMD: Use "Single Instruction, Multiple Data" to parallelize arithmetic at the hardware level.

See references/compute_optimization.md for detailed implementation patterns.

Part 4: Hiding Latency (Perceived Speed)

When you can't make it faster, make it feel faster by masking delays.

• Asynchronous Processing (Pattern: Don't Block the Main Thread):
  • Request Hedging: Send the same request to multiple replicas and use the first response to cut tail latency.
  • Request Batching: Group small requests to amortize round-trip and header overhead.
  • Backpressure: Prevents queuing latency by signaling producers to slow down when the system is saturated.
• Predictive Techniques (Pattern: Guess the Future):
  • Prefetching: Load data (Sequential, Spatial, or Semantic based on user intent) before it's explicitly requested.
  • Optimistic Updates: Update the UI immediately assuming success; reconcile or rollback if the server fails.
• Speculative Execution (Pattern: Execute Before Needed):
  • Parallel Speculation: Execute multiple possible outcomes in parallel and keep the correct one (e.g., in a search engine).
  • Prewarming: Spin up resources (Lambdas, VM instances) based on historical traffic patterns before they are needed.

See references/hiding_latency.md for detailed masking strategies.

Bundled Resources

Scripts

Use these for diagnostics and quick calculations:

• scripts/diagnose_latency.py: Interactive diagnostic checklist runner
• scripts/latency_constants.py: Latency constants for ballpark calculations

Code Examples

See code-examples/ for implementations of key techniques from the book.

Latency Constants (Quick Reference)

Human Perception