Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (Part 1 – Architecture)

At Yelp, Kafka processes hundreds of billions of messages daily to support data‑driven pipelines and critical services. The company recently improved its Kafka deployment architecture by running clusters on PaaSTA, Yelp’s internal platform‑as‑a‑service, leveraging a custom Kubernetes operator and Cruise Control for lifecycle management.

Motivation and Architectural Improvements

Previously, all Kafka clusters ran on dedicated EC2 instances in AWS, with configuration managed via a Puppet repository, a cumbersome process that took over two hours to create a new cluster. The new deployment model aims to reduce reliance on slow Puppet runs, promote internal adoption of PaaSTA, improve maintainability of lifecycle management, simplify OS host and Kafka version upgrades, streamline new cluster creation, and accelerate broker decommissioning while saving resources.

Yelp had already built practices for running stateful workloads such as Cassandra and Flink on Kubernetes, making PaaSTA a natural choice for Kafka.

The new architecture uses PaaSTA pools as the underlying infrastructure. Kafka broker pods are scheduled on Kubernetes nodes and have detachable EBS volumes. Two key components are the Kafka operator and Cruise Control, each deployed per cluster on PaaSTA.

Key differences from the old architecture include running Kafka in Docker containers and moving configuration management away from Puppet to a PaaSTA‑based solution where YAML changes trigger Jenkins‑driven updates. This enables faster cluster provisioning—new Kafka clusters can now be created in half the time.

example-test-prod:
  deploy_group: prod.everything
  pool: kafka
  brokers: 15
  cpus: 5.7  # CPU unit reservation breakdown: (5.7 (kafka) + 0.1 (hacheck) + 0.1 (sensu)) + 0.1 (kiam) = 6.0
  mem: 26Gi
  data: 910Gi
  storage_class: gp2
  cluster_type: example
  cluster_name: test-prod
  use_cruise_control: true
  cruise_control_port: 12345
  service_name: kafka-2-4-1
  zookeeper:
    cluster_name: test-prod
    chroot: kafka-example-test-prod
    cluster_type: kafka_example_test
  config:
    unclean.leader.election.enable: "false"
    reserved.broker.max.id: "2113929216"
    request.timeout.ms: "300001"
    replica.fetch.max.bytes: "10485760"
    offsets.topic.segment.bytes: "104857600"
    offsets.retention.minutes: "10080"
    offsets.load.buffer.size: "15728640"
    num.replica.fetchers: "3"
    num.network.threads: "5"
    num.io.threads: "5"
    min.insync.replicas: "2"
    message.max.bytes: "1000000"
    log.segment.bytes: "268435456"
    log.roll.jitter.hours: "1"
    log.roll.hours: "22"
    log.retention.hours: "24"
    log.message.timestamp.type: "LogAppendTime"
    log.message.format.version: "2.4-IV1"
    log.cleaner.enable: "true"
    log.cleaner.threads: "3"
    log.cleaner.dedupe.buffer.size: "536870912"
    inter.broker.protocol.version: "2.4-IV1"
    group.max.session.timeout.ms: "300000"
    delete.topic.enable: "true"
    default.replication.factor: "3"
    connections.max.idle.ms: "3600000"
    confluent.support.metrics.enable: "false"
    auto.create.topics.enable: "false"
    transactional.id.expiration.ms: "86400000"

Detailed New Architecture

The core of the new design is a custom Kafka Kubernetes operator that manages the state of Kafka clusters. While an external ZooKeeper ensemble still stores metadata, Kafka broker pods retain message data on attached disks, making Kafka a stateful application in Kubernetes.

Kubernetes provides StatefulSets for managing stateful workloads, but it lacks Kafka‑specific constructs, so the operator acts as a custom controller exposing the necessary functionality.

The operator runs a reconciliation loop that watches custom KafkaCluster resources, interacts with the Kubernetes API, and calls the Cruise Control API to perform corrective actions.

Cruise Control, an open‑source Kafka cluster management system from LinkedIn, reduces the operational overhead of large clusters. Each Kafka cluster has its own Cruise Control instance, and the operator uses its API to perform health checks, rebalance partitions, and add or remove brokers.

Both the operator and Cruise Control follow similar paradigms: monitoring cluster state, building an internal model, detecting anomalies, and invoking remediation actions via APIs, replacing the previous ad‑hoc lifecycle scripts that relied on EC2, SNS, and SQS.

Combined, these components form a complete architecture: a Custom Resource Definition (CRD) describes the desired KafkaCluster, the operator creates Kafka broker pods using a custom Docker image, and Cruise Control ensures the cluster remains healthy. Users can observe and interact with the cluster via the Cruise Control UI or PaaSTA CLI.

A typical scaling‑down scenario illustrates the workflow: a developer reduces the broker count in the CRD, the operator detects the drift between desired and actual state, requests Cruise Control to remove brokers, monitors the decommissioning process, and finally deletes the pods once the task completes.

What Comes Next?

After finalizing the architecture, Yelp built a migration process to move Kafka clusters from EC2 to PaaSTA seamlessly. Many clusters have already been migrated, and the team continues to fine‑tune hardware selections to match varying workload characteristics.

The next article will detail the migration strategy for moving existing EC2‑based Kafka clusters onto the Kubernetes‑based internal compute platform.