ML/CV Specialist
Provides specialized guidance for machine learning and computer vision system design, model selection, and production deployment.
When to Use
- Selecting ML models for specific use cases
- Designing training and inference pipelines
- Optimizing ML system performance and cost
- Evaluating build vs. API for ML capabilities
- Planning data pipelines for ML workloads
ML System Design Framework
Model Selection Decision Tree
Use Case Identified
│
├─► Text/Language Tasks
│ ├─► Classification → BERT, DistilBERT, or API (OpenAI, Claude)
│ ├─► Generation → GPT-4, Claude, Llama (self-hosted)
│ ├─► Embeddings → OpenAI Ada, sentence-transformers
│ └─► Search/RAG → Vector DB + Embeddings + LLM
│
├─► Computer Vision Tasks
│ ├─► Classification → ResNet, EfficientNet, ViT
│ ├─► Object Detection → YOLOv8, DETR, Faster R-CNN
│ ├─► Segmentation → SAM, Mask R-CNN, U-Net
│ ├─► OCR → Tesseract, PaddleOCR, Cloud Vision API
│ └─► Face Recognition → InsightFace, DeepFace
│
├─► Audio Tasks
│ ├─► Speech-to-Text → Whisper, DeepSpeech, Cloud APIs
│ ├─► Text-to-Speech → ElevenLabs, Coqui TTS
│ └─► Audio Classification → PANNs, AudioSet models
│
└─► Structured Data
├─► Tabular → XGBoost, LightGBM, CatBoost
├─► Time Series → Prophet, ARIMA, Transformer-based
└─► Recommendations → Two-tower, matrix factorization
API vs. Self-Hosted Decision
When to Use APIs
| Factor | API Preferred | Self-Hosted Preferred |
|---|---|---|
| Volume | < 10K requests/month | > 100K requests/month |
| Latency | > 500ms acceptable | < 100ms required |
| Customization | General use case | Domain-specific fine-tuning |
| Data Privacy | Non-sensitive data | PII, HIPAA, financial |
| Team Expertise | No ML engineers | ML team available |
| Budget | Predictable per-call costs | High volume justifies infra |
Cost Comparison Framework
## API Costs (Example: OpenAI GPT-4)
- Input: $0.03/1K tokens
- Output: $0.06/1K tokens
- Average request: 500 input + 200 output tokens
- Cost per request: $0.027
- 100K requests/month: $2,700
## Self-Hosted Costs (Example: Llama 70B)
- GPU instance: $3/hour (A100 40GB)
- Throughput: ~50 requests/minute = 3K/hour
- Cost per request: $0.001
- 100K requests/month: $100 + $500 engineering time
## Break-even Analysis
- < 50K requests: API likely cheaper
- > 50K requests: Self-hosted may be cheaper
- Factor in: engineering time, ops burden, model quality
Training Pipeline Architecture
Standard ML Pipeline
┌─────────────────────────────────────────────────────────────┐
│ DATA LAYER │
├─────────────────────────────────────────────────────────────┤
│ Data Sources → ETL → Feature Store → Training Data │
│ (S3, DBs) (Airflow) (Feast) (Versioned) │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ TRAINING LAYER │
├─────────────────────────────────────────────────────────────┤
│ Experiment Tracking → Training Jobs → Model Registry │
│ (MLflow, W&B) (SageMaker) (MLflow, S3) │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ SERVING LAYER │
├─────────────────────────────────────────────────────────────┤
│ Model Server → Load Balancer → Monitoring │
│ (TorchServe) (K8s/ELB) (Prometheus) │
└─────────────────────────────────────────────────────────────┘
Component Selection Guide
| Component | Options | Recommendation |
|---|---|---|
| Feature Store | Feast, Tecton, SageMaker | Feast (open source), Tecton (enterprise) |
| Experiment Tracking | MLflow, Weights & Biases, Neptune | MLflow (free), W&B (best UX) |
| Training Orchestration | Kubeflow, SageMaker, Vertex AI | SageMaker (AWS), Vertex (GCP) |
| Model Registry | MLflow, SageMaker, custom S3 | MLflow (standard) |
| Model Serving | TorchServe, TFServing, Triton | Triton (multi-framework) |
Inference Architecture Patterns
Pattern 1: Synchronous API
Best for: Low-latency requirements, simple integration
Client → API Gateway → Model Server → Response
│
Load Balancer
│
┌──────┴──────┐
│ │
Model Pod Model Pod
Latency targets:
- P50: < 100ms
- P95: < 300ms
- P99: < 500ms
Pattern 2: Asynchronous Processing
Best for: Long-running inference, batch processing
Client → API → Queue (SQS) → Worker → Result Store → Webhook/Poll
│
S3/Redis
Use when:
- Inference > 5 seconds
- Batch processing required
- Variable load patterns
Pattern 3: Edge Inference
Best for: Privacy, offline capability, ultra-low latency
┌─────────────────────────────────────────┐
│ EDGE DEVICE │
│ ┌─────────┐ ┌─────────────────────┐ │
│ │ Camera │───▶│ Optimized Model │ │
│ └─────────┘ │ (ONNX, TFLite) │ │
│ └─────────────────────┘ │
│ │ │
│ Local Result │
└─────────────────────────────────────────┘
│
Sync to Cloud
(non-blocking)
Model optimization for edge:
- Quantization (INT8): 4x smaller, 2-3x faster
- Pruning: 50-90% sparsity possible
- Distillation: Smaller model, similar accuracy
- ONNX/TFLite: Optimized runtime
Computer Vision Pipeline Design
Real-Time Video Processing
Camera Stream → Frame Extraction → Preprocessing → Model → Postprocessing → Output
│ │ │ │ │
RTSP/ 1-30 FPS Resize, Batch or NMS, tracking,
WebRTC normalize single annotation
Performance optimization:
- Process every Nth frame (skip frames)
- Resize to model input size early
- Batch frames when latency allows
- Use GPU preprocessing (NVIDIA DALI)
Object Detection System
## Pipeline Components
1. **Input Processing**
- Video decode: FFmpeg, OpenCV
- Frame buffer: Ring buffer for temporal context
- Preprocessing: NVIDIA DALI (GPU), OpenCV (CPU)
2. **Detection**
- Model: YOLOv8 (speed), DETR (accuracy)
- Batch size: 1-8 depending on latency requirements
- Confidence threshold: 0.5-0.7 typical
3. **Post-processing**
- NMS (Non-Maximum Suppression)
- Tracking: SORT, DeepSORT, ByteTrack
- Smoothing: Kalman filter for stable boxes
4. **Output**
- Annotations: Bounding boxes, labels, confidence
- Events: Trigger on detection (webhook, queue)
- Storage: Frame + metadata to S3/DB
LLM Integration Patterns
RAG (Retrieval-Augmented Generation)
User Query → Embedding → Vector Search → Context Retrieval → LLM → Response
│
Vector DB
(Pinecone, Weaviate,
Chroma, pgvector)
Vector DB Selection:
| Database | Best For | Limitations |
|---|---|---|
| Pinecone | Managed, scale | Cost at scale |
| Weaviate | Self-hosted, features | Operational overhead |
| Chroma | Simple, local dev | Not for production scale |
| pgvector | PostgreSQL users | Performance at >1M vectors |
| Qdrant | Performance | Newer, smaller community |
LLM Serving Architecture
┌─────────────────────────────────────────────────────────────┐
│ API GATEWAY │
│ Rate limiting, auth, request routing │
└─────────────────────────────────────────────────────────────┘
│
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────┐
│ GPT-4 │ │ Claude │ │ Local │
│ API │ │ API │ │ Llama │
└────────┘ └────────┘ └────────┘
│
Model Router
(cost/latency/capability)
Multi-model strategy:
- Simple queries → Cheaper model (GPT-3.5, Haiku)
- Complex reasoning → Expensive model (GPT-4, Opus)
- Sensitive data → Self-hosted (Llama, Mistral)
Performance Optimization
GPU Memory Optimization
| Technique | Memory Reduction | Speed Impact |
|---|---|---|
| FP16 (Half Precision) | 50% | Neutral to faster |
| INT8 Quantization | 75% | 10-20% slower |
| INT4 Quantization | 87.5% | 20-40% slower |
| Gradient Checkpointing | 60-80% | 20-30% slower |
| Model Sharding | Distributed | Communication overhead |
Batching Strategies
# Dynamic batching pseudocode
class DynamicBatcher:
def __init__(self, max_batch=32, max_wait_ms=50):
self.queue = []
self.max_batch = max_batch
self.max_wait = max_wait_ms
async def add_request(self, request):
self.queue.append(request)
# Batch when full or timeout
if len(self.queue) >= self.max_batch:
return await self.process_batch()
await asyncio.sleep(self.max_wait / 1000)
return await self.process_batch()
async def process_batch(self):
batch = self.queue[:self.max_batch]
self.queue = self.queue[self.max_batch:]
return await self.model.predict_batch(batch)
Model Monitoring
Key Metrics to Track
| Metric | What It Measures | Alert Threshold |
|---|---|---|
| Latency (P95) | Response time | > 2x baseline |
| Throughput | Requests/second | < 80% capacity |
| Error Rate | Failed predictions | > 1% |
| Model Drift | Distribution shift | PSI > 0.2 |
| Data Quality | Input anomalies | > 5% anomalies |
Drift Detection
Training Distribution ──┐
├──► Statistical Test ──► Alert
Production Distribution ─┘
(PSI, KS test, JS divergence)
Population Stability Index (PSI):
- PSI < 0.1: No significant change
- 0.1 < PSI < 0.2: Moderate change, monitor
- PSI > 0.2: Significant change, investigate
Quick Reference Tables
Model Selection by Use Case
| Use Case | Recommended Model | Latency | Cost |
|---|---|---|---|
| Text Classification | DistilBERT | 10ms | Low |
| Text Generation | GPT-4 / Claude | 1-5s | Medium |
| Image Classification | EfficientNet-B0 | 5ms | Low |
| Object Detection | YOLOv8-n | 10ms | Low |
| Object Detection (Accurate) | YOLOv8-x | 50ms | Medium |
| Semantic Segmentation | SAM | 100ms | Medium |
| Speech-to-Text | Whisper-base | Real-time | Low |
| Embeddings | text-embedding-ada-002 | 50ms | Low |
Infrastructure Sizing
| Scale | GPU | Model Size | Throughput |
|---|---|---|---|
| Development | T4 (16GB) | < 7B params | 10-50 req/s |
| Production Small | A10G (24GB) | < 13B params | 50-100 req/s |
| Production Medium | A100 (40GB) | < 70B params | 100-500 req/s |
| Production Large | A100 (80GB) x 2+ | > 70B params | 500+ req/s |
References
- Model Catalog - Detailed model comparison and benchmarks
- Inference Patterns - Architecture patterns for different use cases