LLM Serving Patterns

When to Use This Skill

Use this skill when:

• Designing LLM inference infrastructure

• Choosing between serving frameworks (vLLM, TGI, TensorRT-LLM)

• Implementing quantization for production deployment

• Optimizing batching and throughput

• Building streaming response systems

• Scaling LLM deployments cost-effectively

Keywords: LLM serving, inference, vLLM, TGI, TensorRT-LLM, quantization, INT8, INT4, FP16, batching, continuous batching, streaming, SSE, WebSocket, KV cache, PagedAttention, speculative decoding

LLM Serving Architecture Overview

┌─────────────────────────────────────────────────────────────────────┐ │ LLM Serving Stack │ ├─────────────────────────────────────────────────────────────────────┤ │ Clients (API, Chat UI, Agents) │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Load Balancer / API Gateway │ │ │ │ • Rate limiting • Authentication • Request routing │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Inference Server │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ │ │ │ │ Request │ │ Batching │ │ KV Cache │ │ │ │ │ │ Queue │──▶│ Engine │──▶│ Management │ │ │ │ │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ │ │ │ │ │ │ │ │ │ ▼ ▼ │ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ │ │ Model Execution Engine │ │ │ │ │ │ • Tensor operations • Attention • Token sampling │ │ │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ GPU/TPU Cluster │ │ │ │ • Model sharding • Tensor parallelism • Pipeline parallel │ │ │ └─────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘

Serving Framework Comparison

Framework Strengths Best For Considerations

vLLM PagedAttention, high throughput, continuous batching General LLM serving, high concurrency Python-native, active community

TGI (Text Generation Inference) Production-ready, Hugging Face integration Enterprise deployment, HF models Rust backend, Docker-first

TensorRT-LLM NVIDIA optimization, lowest latency NVIDIA GPUs, latency-critical NVIDIA-only, complex setup

Triton Inference Server Multi-model, multi-framework Heterogeneous model serving Enterprise complexity

Ollama Simple local deployment Development, edge deployment Limited scaling features

llama.cpp CPU inference, quantization Resource-constrained, edge C++ integration required

Framework Selection Decision Tree

Need lowest latency on NVIDIA GPUs? ├── Yes → TensorRT-LLM └── No └── Need high throughput with many concurrent users? ├── Yes → vLLM (PagedAttention) └── No └── Need enterprise features + HF integration? ├── Yes → TGI └── No └── Simple local/edge deployment? ├── Yes → Ollama or llama.cpp └── No → vLLM (general purpose)

Quantization Techniques

Precision Levels

Precision Bits Memory Reduction Quality Impact Use Case

FP32 32 Baseline None Training, reference

FP16/BF16 16 2x Minimal Standard serving

INT8 8 4x Low Production serving

INT4 4 8x Moderate Resource-constrained

INT2 2 16x Significant Experimental

Quantization Methods

Method Description Quality Speed

PTQ (Post-Training Quantization) Quantize after training, no retraining Good Fast to apply

QAT (Quantization-Aware Training) Simulate quantization during training Better Requires training

GPTQ One-shot weight quantization Very good Moderate

AWQ (Activation-aware Weight Quantization) Preserves salient weights Excellent Moderate

GGUF/GGML llama.cpp format, CPU-optimized Good Very fast inference

SmoothQuant Migrates difficulty to weights Excellent Moderate

Quantization Selection

Quality vs. Efficiency Trade-off:

Quality ────────────────────────────────────────────▶ Efficiency │ │ │ FP32 FP16 INT8+AWQ INT8+GPTQ INT4 INT2 │ │ ○───────○────────○──────────○──────────○──────○ │ │ │ │ │ │ │ │ │ │ Best Great Good Good Fair Poor │ │ │

Batching Strategies

Static Batching

Request 1: [tokens: 100] ─┐ Request 2: [tokens: 50] ─┼──▶ [Batch: pad to 100] ──▶ Process ──▶ All complete Request 3: [tokens: 80] ─┘

Problem: Short requests wait for long ones (head-of-line blocking)

Continuous Batching (Preferred)

Time ──────────────────────────────────────────────────────────▶

Req 1: [████████████████████████████████] ──▶ Complete Req 2: [████████████] ──▶ Complete ──▶ Req 4 starts [████████████████] Req 3: [████████████████████] ──▶ Complete ──▶ Req 5 starts [████████]

• New requests join batch as others complete • No padding waste • Optimal GPU utilization

Batching Parameters

Parameter Description Trade-off

max_batch_size

Maximum concurrent requests Memory vs. throughput

max_waiting_tokens

Tokens before forcing batch Latency vs. throughput

max_num_seqs

Maximum sequences in batch Memory vs. concurrency

KV Cache Management

The KV Cache Problem

Attention: Q × K^T × V

For each token generated: • Must recompute attention with ALL previous tokens • K and V tensors grow with sequence length • Memory: O(batch_size × seq_len × num_layers × hidden_dim)

Example (70B model, 4K context): • KV cache per request: ~8GB • 10 concurrent requests: ~80GB GPU memory

PagedAttention (vLLM Innovation)

Traditional KV Cache: ┌──────────────────────────────────────────┐ │ Request 1 KV Cache (contiguous, fixed) │ ← Wastes memory ├──────────────────────────────────────────┤ │ Request 2 KV Cache (contiguous, fixed) │ ├──────────────────────────────────────────┤ │ FRAGMENTED/WASTED SPACE │ └──────────────────────────────────────────┘

PagedAttention: ┌────┬────┬────┬────┬────┬────┬────┬────┐ │ R1 │ R2 │ R1 │ R3 │ R2 │ R1 │ R3 │ R2 │ ← Pages allocated on demand └────┴────┴────┴────┴────┴────┴────┴────┘ • Non-contiguous memory allocation • Near-zero memory waste • 2-4x higher throughput

KV Cache Optimization Strategies

Strategy Description Memory Savings

Paged Attention Virtual memory for KV cache ~50% reduction

Prefix Caching Reuse KV cache for common prefixes System prompt: 100%

Quantized KV Cache INT8/FP8 for KV values 50-75% reduction

Sliding Window Limited attention context Linear memory

MQA/GQA Grouped query attention Architecture-dependent

Streaming Response Patterns

Server-Sent Events (SSE)

Client Server │ │ │──── GET /v1/chat/completions ──────▶│ │ (stream: true) │ │ │ │◀──── HTTP 200 OK ───────────────────│ │ Content-Type: text/event-stream│ │ │ │◀──── data: {"token": "Hello"} ──────│ │◀──── data: {"token": " world"} ─────│ │◀──── data: {"token": "!"} ──────────│ │◀──── data: [DONE] ──────────────────│ │ │

SSE Benefits:

• HTTP/1.1 compatible

• Auto-reconnection support

• Simple to implement

• Wide client support

WebSocket Streaming

Client Server │ │ │──── WebSocket Upgrade ─────────────▶│ │◀──── 101 Switching Protocols ───────│ │ │ │──── {"prompt": "Hello"} ───────────▶│ │ │ │◀──── {"token": "Hi"} ───────────────│ │◀──── {"token": " there"} ───────────│ │◀──── {"token": "!"} ────────────────│ │◀──── {"done": true} ────────────────│ │ │

WebSocket Benefits:

• Bidirectional communication

• Lower latency

• Better for chat applications

• Connection persistence

Streaming Implementation Considerations

Aspect SSE WebSocket

Reconnection Built-in Manual

Scalability Per-request Connection pool

Load Balancing Standard HTTP Sticky sessions

Firewall/Proxy Usually works May need config

Best For One-way streaming Interactive chat

Speculative Decoding

Concept

Standard Decoding: Large Model: [T1] → [T2] → [T3] → [T4] → [T5] 10ms 10ms 10ms 10ms 10ms = 50ms total

Speculative Decoding: Draft Model: [T1, T2, T3, T4, T5] (parallel, 5ms) │ ▼ Large Model: [Verify T1-T5 in one pass] (15ms) Accept: T1, T2, T3 ✓ Reject: T4, T5 ✗ │ ▼ [Generate T4, T5 correctly]

Total: ~25ms (2x speedup if 60% acceptance)

Speculative Decoding Trade-offs

Factor Impact

Draft model quality Higher match rate = more speedup

Draft model size Larger = better quality, slower

Speculation depth More tokens = higher risk/reward

Verification cost Must be < sequential generation

Scaling Strategies

Horizontal Scaling

┌─────────────────────────────────────────────────────────┐ │ Load Balancer │ │ (Round-robin, Least-connections) │ └─────────────────────────────────────────────────────────┘ │ │ │ ▼ ▼ ▼ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ vLLM │ │ vLLM │ │ vLLM │ │ Node 1 │ │ Node 2 │ │ Node 3 │ │ (GPU×4) │ │ (GPU×4) │ │ (GPU×4) │ └─────────┘ └─────────┘ └─────────┘

Model Parallelism

Strategy Description Use Case

Tensor Parallelism Split layers across GPUs Single large model

Pipeline Parallelism Different layers on different GPUs Very large models

Data Parallelism Same model, different batches High throughput

Tensor Parallelism (TP=4): ┌─────────────────────────────────────────┐ │ Layer N │ │ GPU0 │ GPU1 │ GPU2 │ GPU3 │ │ 25% │ 25% │ 25% │ 25% │ └─────────────────────────────────────────┘

Pipeline Parallelism (PP=4): GPU0: Layers 0-7 GPU1: Layers 8-15 GPU2: Layers 16-23 GPU3: Layers 24-31

Latency Optimization Checklist

Pre-deployment

• Choose appropriate quantization (INT8 for production)

• Enable continuous batching

• Configure KV cache size appropriately

• Set optimal batch size for hardware

• Enable prefix caching for system prompts

Runtime

• Monitor GPU memory utilization

• Track p50/p95/p99 latencies

• Measure time-to-first-token (TTFT)

• Monitor tokens-per-second (TPS)

• Set appropriate timeouts

Infrastructure

• Use fastest available interconnect (NVLink, InfiniBand)

• Minimize network hops

• Place inference close to users (edge)

• Consider dedicated inference hardware

Cost Optimization

Cost Drivers

Factor Impact Optimization

GPU hours Highest Quantization, batching

Memory High PagedAttention, KV cache optimization

Network Medium Response compression, edge deployment

Storage Low Model deduplication

Cost Estimation Formula

Monthly Cost = (Requests/month) × (Avg tokens/request) × (GPU-seconds/token) × ($/GPU-hour) ───────────────────────────────────────────────────────────────────────────── 3600

Example: • 10M requests/month • 500 tokens average • 0.001 GPU-seconds/token (optimized) • $2/GPU-hour

Cost = (10M × 500 × 0.001 × 2) / 3600 = $2,778/month

Common Patterns

Multi-model Routing

┌─────────────────────────────────────────────────────────┐ │ Router │ │ • Classify request complexity │ │ • Route to appropriate model │ └─────────────────────────────────────────────────────────┘ │ │ │ ▼ ▼ ▼ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ Small │ │ Medium │ │ Large │ │ Model │ │ Model │ │ Model │ │ (7B) │ │ (13B) │ │ (70B) │ │ Fast │ │ Balanced│ │ Quality │ └─────────┘ └─────────┘ └─────────┘

Caching Strategies

Cache Type What to Cache TTL

Prompt cache Common system prompts Long

KV cache Prefix tokens Session

Response cache Exact query matches Varies

Embedding cache Document embeddings Long

Related Skills

• ml-system-design

• End-to-end ML pipeline design

• rag-architecture

• Retrieval-augmented generation patterns

• vector-databases

• Vector search for LLM context

• ml-inference-optimization

• General inference optimization

• estimation-techniques

• Capacity planning for LLM systems

Version History

• v1.0.0 (2025-12-26): Initial release - LLM serving patterns for systems design interviews

Last Updated

Date: 2025-12-26