Kafka Operations

Expert in Apache Kafka deployment, monitoring, and operational tooling.

⚠️ Chunking Rule

Large Kafka infrastructure = 800+ lines. Generate ONE component per response:

• Deployment → 2. Monitoring → 3. CLI Tools → 4. Automation

Core Capabilities

Kubernetes Deployment

• Strimzi Operator: Open-source Kafka on K8s

• Confluent for Kubernetes: Enterprise Kafka

• MSK/Confluent Cloud: Managed services

Infrastructure as Code

• Terraform modules for Kafka clusters

• AWS MSK, Confluent Cloud, Aiven provisioning

• Network and security configuration

Observability

• Prometheus metrics (JMX exporter)

• Grafana dashboards for Kafka

• Consumer lag monitoring

• Alert configuration

CLI Tools

• kcat: Swiss army knife for Kafka

• kafkactl: Modern CLI for Kafka

• kafka-console-*: Built-in tools

Kubernetes Deployment

Strimzi Kafka Cluster

apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: replicas: 3 listeners: - name: plain port: 9092 type: internal tls: false storage: type: persistent-claim size: 100Gi zookeeper: replicas: 3 storage: type: persistent-claim size: 50Gi

Terraform

AWS MSK Cluster

resource "aws_msk_cluster" "kafka" { cluster_name = "my-kafka-cluster" kafka_version = "3.5.1" number_of_broker_nodes = 3

broker_node_group_info { instance_type = "kafka.m5.large" client_subnets = var.private_subnets security_groups = [aws_security_group.kafka.id] storage_info { ebs_storage_info { volume_size = 100 } } } }

Monitoring

Prometheus scrape config

• job_name: 'kafka' static_configs:
  • targets: ['kafka-1:9404', 'kafka-2:9404', 'kafka-3:9404'] relabel_configs:
  • source_labels: [address] target_label: instance

Key metrics to monitor:

• kafka_server_brokertopicmetrics_messagesin_total

• kafka_consumer_consumer_fetch_manager_metrics_records_lag

• kafka_server_replicamanager_underreplicatedpartitions

CLI Examples

kcat - produce message

echo '{"event":"order.created"}' | kcat -P -b localhost:9092 -t orders

kcat - consume messages

kcat -C -b localhost:9092 -t orders -o beginning

kafkactl - describe topic

kafkactl describe topic orders

kafkactl - consumer groups

kafkactl get consumer-groups kafkactl describe consumer-group order-processor

When to Use

• Deploying Kafka on Kubernetes

• Setting up Kafka with Terraform

• Configuring monitoring and alerts

• Operational tasks with CLI tools

• Troubleshooting Kafka issues