Kafka Architecture & Design Expert

Comprehensive knowledge of Apache Kafka architecture patterns, cluster design principles, and production best practices for building resilient, scalable event streaming platforms.

Core Architecture Concepts

Kafka Cluster Components

Brokers:

• Individual Kafka servers that store and serve data

• Each broker handles thousands of partitions

• Typical: 3-10 brokers per cluster (small), 10-100+ (large enterprises)

Controller:

• One broker elected as controller (via KRaft or ZooKeeper)

• Manages partition leaders and replica assignments

• Failure triggers automatic re-election

Topics:

• Logical channels for message streams

• Divided into partitions for parallelism

• Can have different retention policies per topic

Partitions:

• Ordered, immutable sequence of records

• Unit of parallelism (1 partition = 1 consumer in a group)

• Distributed across brokers for load balancing

Replicas:

• Copies of partitions across multiple brokers

• 1 leader replica (serves reads/writes)

• N-1 follower replicas (replication only)

• In-Sync Replicas (ISR): Followers caught up with leader

KRaft vs ZooKeeper Mode

KRaft Mode (Recommended, Kafka 3.3+):

Cluster Metadata:

• Stored in Kafka itself (no external ZooKeeper)
• Metadata topic: __cluster_metadata
• Controller quorum (3 or 5 nodes)
• Faster failover (<1s vs 10-30s)
• Simplified operations

ZooKeeper Mode (Legacy, deprecated in 4.0):

External Coordination:

• Requires separate ZooKeeper ensemble (3-5 nodes)
• Stores cluster metadata, configs, ACLs
• Slower failover (10-30 seconds)
• More complex to operate

Migration: ZooKeeper → KRaft migration supported in Kafka 3.6+

Cluster Sizing Guidelines

Small Cluster (Development/Testing)

Configuration: Brokers: 3 Partitions per broker: ~100-500 Total partitions: 300-1500 Replication factor: 3 Hardware: - CPU: 4-8 cores - RAM: 8-16 GB - Disk: 500 GB - 1 TB SSD - Network: 1 Gbps

Use Cases:

• Development environments
• Low-volume production (<10 MB/s)
• Proof of concepts
• Single datacenter

Example Workload:

• 50 topics
• 5-10 partitions per topic
• 1 million messages/day
• 7-day retention

Medium Cluster (Standard Production)

Configuration: Brokers: 6-12 Partitions per broker: 500-2000 Total partitions: 3K-24K Replication factor: 3 Hardware: - CPU: 16-32 cores - RAM: 64-128 GB - Disk: 2-8 TB NVMe SSD - Network: 10 Gbps

Use Cases:

• Standard production workloads
• Multi-team environments
• Regional deployments
• Up to 500 MB/s throughput

Example Workload:

• 200-500 topics
• 10-50 partitions per topic
• 100 million messages/day
• 30-day retention

Large Cluster (High-Scale Production)

Configuration: Brokers: 20-100+ Partitions per broker: 2000-4000 Total partitions: 40K-400K+ Replication factor: 3 Hardware: - CPU: 32-64 cores - RAM: 128-256 GB - Disk: 8-20 TB NVMe SSD - Network: 25-100 Gbps

Use Cases:

• Large enterprises
• Multi-region deployments
• Event-driven architectures
• 1+ GB/s throughput

Example Workload:

• 1000+ topics
• 50-200 partitions per topic
• 1+ billion messages/day
• 90-365 day retention

Kafka Streams / Exactly-Once Semantics (EOS) Clusters

Configuration: Brokers: 6-12+ (same as standard, but more control plane load) Partitions per broker: 500-1500 (fewer due to transaction overhead) Total partitions: 3K-18K Replication factor: 3 Hardware: - CPU: 16-32 cores (more CPU for transactions) - RAM: 64-128 GB - Disk: 4-12 TB NVMe SSD (more for transaction logs) - Network: 10-25 Gbps

Special Considerations:

• More brokers due to transaction coordinator load
• Lower partition count per broker (transactions = more overhead)
• Higher disk IOPS for transaction logs
• min.insync.replicas=2 mandatory for EOS
• acks=all required for producers

Use Cases:

• Stream processing with exactly-once guarantees
• Financial transactions
• Event sourcing with strict ordering
• Multi-step workflows requiring atomicity

Partitioning Strategy

How Many Partitions?

Formula:

Partitions = max( Target Throughput / Single Partition Throughput, Number of Consumers (for parallelism), Future Growth Factor (2-3x) )

Single Partition Limits:

• Write throughput: ~10-50 MB/s
• Read throughput: ~30-100 MB/s
• Message rate: ~10K-100K msg/s

Examples:

High Throughput Topic (Logs, Events):

Requirements:

• Write: 200 MB/s
• Read: 500 MB/s (multiple consumers)
• Expected growth: 3x in 1 year

Calculation: Write partitions: 200 MB/s ÷ 20 MB/s = 10 Read partitions: 500 MB/s ÷ 40 MB/s = 13 Growth factor: 13 × 3 = 39

Recommendation: 40-50 partitions

Low-Latency Topic (Commands, Requests):

Requirements:

• Write: 5 MB/s
• Read: 10 MB/s
• Latency: <10ms p99
• Order preservation: By user ID

Calculation: Throughput partitions: 5 MB/s ÷ 20 MB/s = 1 Parallelism: 4 (for redundancy)

Recommendation: 4-6 partitions (keyed by user ID)

Dead Letter Queue:

Recommendation: 1-3 partitions Reason: Low volume, order less important

Partition Key Selection

Good Keys (High Cardinality, Even Distribution):

✅ User ID (UUIDs):

• Millions of unique values
• Even distribution
• Example: "user-123e4567-e89b-12d3-a456-426614174000"

✅ Device ID (IoT):

• Unique per device
• Natural sharding
• Example: "device-sensor-001-zone-a"

✅ Order ID (E-commerce):

• Unique per transaction
• Even temporal distribution
• Example: "order-2024-11-15-abc123"

Bad Keys (Low Cardinality, Hotspots):

❌ Country Code:

• Only ~200 values
• Uneven (US, CN >> others)
• Creates partition hotspots

❌ Boolean Flags:

• Only 2 values (true/false)
• Severe imbalance

❌ Date (YYYY-MM-DD):

• All today's traffic → 1 partition
• Temporal hotspot

Compound Keys (Best of Both):

✅ Country + User ID:

• Partition by country for locality
• Sub-partition by user for distribution
• Example: "US:user-123" → hash("US:user-123")

✅ Tenant + Event Type + Timestamp:

• Multi-tenant isolation
• Event type grouping
• Temporal ordering

Replication & High Availability

Replication Factor Guidelines

Development: Replication Factor: 1 Reason: Fast, no durability needed

Production (Standard): Replication Factor: 3 Reason: Balance durability vs cost Tolerates: 2 broker failures (with min.insync.replicas=2)

Production (Critical): Replication Factor: 5 Reason: Maximum durability Tolerates: 4 broker failures (with min.insync.replicas=3) Use Cases: Financial transactions, audit logs

Multi-Datacenter: Replication Factor: 3 per DC (6 total) Reason: DC-level fault tolerance Requires: MirrorMaker 2 or Confluent Replicator

min.insync.replicas

Configuration:

min.insync.replicas=2:

• At least 2 replicas must acknowledge writes
• Typical for replication.factor=3
• Prevents data loss if 1 broker fails

min.insync.replicas=1:

• Only leader must acknowledge (dangerous!)
• Use only for non-critical topics

min.insync.replicas=3:

• At least 3 replicas must acknowledge
• For replication.factor=5 (critical systems)

Rule: min.insync.replicas ≤ replication.factor - 1 (to allow 1 replica failure)

Rack Awareness

Configuration: broker.rack=rack1 # Broker 1 broker.rack=rack2 # Broker 2 broker.rack=rack3 # Broker 3

Benefit:

• Replicas spread across racks
• Survives rack-level failures (power, network)
• Example: Topic with RF=3 → 1 replica per rack

Placement: Leader: rack1 Follower 1: rack2 Follower 2: rack3

Retention Strategies

Time-Based Retention

Short-Term (Events, Logs): retention.ms: 86400000 # 1 day Use Cases: Real-time analytics, monitoring

Medium-Term (Transactions): retention.ms: 604800000 # 7 days Use Cases: Standard business events

Long-Term (Audit, Compliance): retention.ms: 31536000000 # 365 days Use Cases: Regulatory requirements, event sourcing

Infinite (Event Sourcing): retention.ms: -1 # Forever cleanup.policy: compact Use Cases: Source of truth, state rebuilding

Size-Based Retention

retention.bytes: 10737418240 # 10 GB per partition

Combined (Time OR Size): retention.ms: 604800000 # 7 days retention.bytes: 107374182400 # 100 GB

Whichever limit is reached first

Compaction (Log Compaction)

cleanup.policy: compact

How It Works:

• Keeps only latest value per key
• Deletes old versions
• Preserves full history initially, compacts later

Use Cases:

• Database changelogs (CDC)
• User profile updates
• Configuration management
• State stores

Example: Before Compaction: user:123 → {name: "Alice", v:1} user:123 → {name: "Alice", v:2, email: "alice@ex.com"} user:123 → {name: "Alice A.", v:3}

After Compaction: user:123 → {name: "Alice A.", v:3} # Latest only

Performance Optimization

Broker Configuration

Network threads (handle client connections)

num.network.threads: 8 # Increase for high connection count

I/O threads (disk operations)

num.io.threads: 16 # Set to number of disks × 2

Replica fetcher threads

num.replica.fetchers: 4 # Increase for many partitions

Socket buffer sizes

socket.send.buffer.bytes: 1048576 # 1 MB socket.receive.buffer.bytes: 1048576 # 1 MB

Log flush (default: OS handles flushing)

log.flush.interval.messages: 10000 # Flush every 10K messages log.flush.interval.ms: 1000 # Or every 1 second

Producer Optimization

High Throughput: batch.size: 65536 # 64 KB linger.ms: 100 # Wait 100ms for batching compression.type: lz4 # Fast compression acks: 1 # Leader only

Low Latency: batch.size: 16384 # 16 KB (default) linger.ms: 0 # Send immediately compression.type: none acks: 1

Durability (Exactly-Once): batch.size: 16384 linger.ms: 10 compression.type: lz4 acks: all enable.idempotence: true transactional.id: "producer-1"

Consumer Optimization

High Throughput: fetch.min.bytes: 1048576 # 1 MB fetch.max.wait.ms: 500 # Wait 500ms to accumulate

Low Latency: fetch.min.bytes: 1 # Immediate fetch fetch.max.wait.ms: 100 # Short wait

Max Parallelism:

Deploy consumers = number of partitions

More consumers than partitions = idle consumers

Multi-Datacenter Patterns

Active-Passive (Disaster Recovery)

Architecture: Primary DC: Full Kafka cluster Secondary DC: Replica cluster (MirrorMaker 2)

Configuration:

• Producers → Primary only
• Consumers → Primary only
• MirrorMaker 2: Primary → Secondary (async replication)

Failover:

1. Detect primary failure
2. Switch producers/consumers to secondary
3. Promote secondary to primary

Recovery Time: 5-30 minutes (manual) Data Loss: Potential (async replication lag)

Active-Active (Geo-Replication)

Architecture: DC1: Kafka cluster (region A) DC2: Kafka cluster (region B) Bidirectional replication via MirrorMaker 2

Configuration:

• Producers → Nearest DC
• Consumers → Nearest DC or both
• Conflict resolution: Last-write-wins or custom

Challenges:

• Duplicate messages (at-least-once delivery)
• Ordering across DCs not guaranteed
• Circular replication prevention

Use Cases:

• Global applications
• Regional compliance (GDPR)
• Load distribution

Stretch Cluster (Synchronous Replication)

Architecture: Single Kafka cluster spanning 2 DCs Rack awareness: DC1 = rack1, DC2 = rack2

Configuration: min.insync.replicas: 2 replication.factor: 4 (2 per DC) acks: all

Requirements:

• Low latency between DCs (<10ms)
• High bandwidth link (10+ Gbps)
• Dedicated fiber

Trade-offs: Pros: Synchronous replication, zero data loss Cons: Latency penalty, network dependency

Monitoring & Observability

Key Metrics

Broker Metrics:

UnderReplicatedPartitions: Alert: > 0 for > 5 minutes Indicates: Replica lag, broker failure

OfflinePartitionsCount: Alert: > 0 Indicates: No leader elected (critical!)

ActiveControllerCount: Alert: != 1 (should be exactly 1) Indicates: Split brain or no controller

RequestHandlerAvgIdlePercent: Alert: < 20% Indicates: Broker CPU saturation

Topic Metrics:

MessagesInPerSec: Monitor: Throughput trends Alert: Sudden drops (producer failure)

BytesInPerSec / BytesOutPerSec: Monitor: Network utilization Alert: Approaching NIC limits

RecordsLagMax (Consumer): Alert: > 10000 or growing Indicates: Consumer can't keep up

Disk Metrics:

LogSegmentSize: Monitor: Disk usage trends Alert: > 80% capacity

LogFlushRateAndTimeMs: Monitor: Disk write latency Alert: > 100ms p99 (slow disk)

Security Patterns

Authentication & Authorization

SASL/SCRAM-SHA-512:

• Industry standard
• User/password authentication
• Stored in ZooKeeper/KRaft

ACLs (Access Control Lists):

• Per-topic, per-group permissions
• Operations: READ, WRITE, CREATE, DELETE, ALTER
• Example: bin/kafka-acls.sh --add
  --allow-principal User:alice
  --operation READ
  --topic orders

mTLS (Mutual TLS):

• Certificate-based auth
• Strong cryptographic identity
• Best for service-to-service

Integration with SpecWeave

Automatic Architecture Detection:

import { ClusterSizingCalculator } from './lib/utils/sizing';

const calculator = new ClusterSizingCalculator(); const recommendation = calculator.calculate({ throughputMBps: 200, retentionDays: 30, replicationFactor: 3, topicCount: 100 });

console.log(recommendation); // { // brokers: 8, // partitionsPerBroker: 1500, // diskPerBroker: 6000 GB, // ramPerBroker: 64 GB // }

SpecWeave Commands:

• /sw-kafka:deploy

• Validates cluster sizing before deployment

• /sw-kafka:monitor-setup

• Configures metrics for key indicators

Related Skills

• /sw-kafka:kafka-mcp-integration

• MCP server setup

• /sw-kafka:kafka-cli-tools

• CLI operations

External Links

• Kafka Documentation - Architecture

• Confluent - Kafka Sizing

• KRaft Mode Overview

• LinkedIn Engineering - Kafka at Scale