Designing Kafka Topics at Ten‑Million QPS: From Ad‑hoc to Standardized Governance

1. From a "test_v2_final" incident

In the early hours of a promotion, the operations team reported that coupon distribution was chaotic: some users received no coupons, others received three duplicates within a second, and customer‑service complaints quadrupled in ten minutes. Monitoring showed a healthy Kafka cluster (stable 80k TPS producer rate, normal consumer lag), but tracing revealed two unrelated services writing to the same topic test_v2_final. Git blame showed the topic was created a year earlier by a marketing engineer for a test, never renamed or deleted, and later reused by a member of the membership team. The two services produced different schemas, partition keys, and consumer groups, causing the chaos when traffic surged.

The root cause was not Kafka or consumer code but the lack of any naming or governance policy for topics—"ad‑hoc" creation had become the norm.

2. Million‑QPS era: ad‑hoc topics can survive

When a company handles a few hundred thousand QPS, the total number of topics is typically 30‑80. Governance relies on informal memory: the creator maintains the topic, deletes it when unused, and naming is free‑form (e.g., order_notify, OrderNotifyV2, prod_order_event, topic_001). Because the topic count is low, everyone can locate a topic via a wiki.

This "good‑enough" approach works only while the number of topics stays small; at ten‑million QPS the cracks appear.

3. Ten‑Million‑QPS transformation: three dimensions explode

3.1 Quantity – from dozens to thousands

At million‑QPS the cluster holds ~50 topics; at ten‑million QPS it expands to 3,000‑10,000 topics. Each business domain, event type, priority, and environment may need its own topic. When the topic count reaches the thousands, broker file‑handle limits, ZooKeeper/KRaft metadata size, and controller election load can cause silent performance collapse.

3.2 Dependency – from single‑service to multi‑service sharing

A topic that once had one producer and one or two consumers now backs 5 producers, 20 consumer groups, and dozens of downstream systems (real‑time data warehouse, risk control, search, recommendation, A/B testing). A single schema change can affect an entire business network.

3.3 Organization – from team‑owned to cross‑team contract

In the million‑QPS stage a topic’s ownership stays within one team. At ten‑million QPS the topic becomes a contract: upstream services produce events, many unrelated downstream teams consume them, and stability, schema compatibility, and SLA can no longer rely on informal verbal sync.

4. Naming convention – from "test_v2" to "{env}.{domain}.{entity}.{event}.{version}"

A cheap but powerful lever is a structured naming scheme. The recommended pattern is: {env}.{domain}.{entity}.{event}.{version} Examples: prod.order.order.created.v1 – production, order domain, order entity, created event, version 1 prod.pay.refund.succeeded.v2 – production, payment domain, refund entity, succeeded event, version 2 stg.user.profile.updated.v1 – staging, user domain, profile entity, updated event, version 1

Each segment has a fixed meaning, enabling permission checks, audit, and automatic routing based on the name alone.

4.2 Avoid over‑loading the name

Bad examples that mix runtime configuration into the name: order_topic_v2_highvolume_no_compression – encodes volume and compression kafka_cluster_a_order_event_retry_v2 – encodes cluster and retry flag test_v2_final_real_final – carries code‑iteration history

These hide essential metadata in the string; instead, keep such information in the topic’s metadata.

4.3 Environment isolation should be done at the cluster level

While adding prefixes like prod_, stg_, or test_ works at million‑QPS, at ten‑million QPS the safe practice is to isolate environments with separate Kafka clusters (production, staging, test/mock). The prefix can remain as a double‑check, but the cluster provides the real safety net against accidental cross‑environment writes.

5. Granularity design – how many event types belong in one topic

5.1 Three granularity styles

Big pot : all related events share one topic; consumers filter by a field.

Entity‑level : events of the same entity share a topic.

Event‑level : each business event gets its own topic.

Each has trade‑offs.

5.2 Three decision rules

If most consumers only care about a subset of events, split the topic.

Events that must be atomically persisted together should stay in the same topic to avoid distributed transactions.

When schema evolution cadence differs greatly between events, keep them separate; otherwise a fast‑changing schema forces all consumers to evolve.

5.3 Empirical split for ten‑million QPS

A healthy distribution is roughly 60% core‑entity topics, 30% single‑event topics, and 10% big‑pot topics (usually logs or audit).

6. Partition planning – Partition Key as an ordered‑budget

6.1 Partition Key purpose

Kafka guarantees order only within a single partition. The key answers “which messages must be strictly ordered?”. Typical keys: order_id for order lifecycle events user_id for user‑behavior tracking conversation_id for IM messages device_id for device telemetry

Finer keys increase concurrency but reduce the amount of ordering guaranteed; coarser keys do the opposite.

6.2 Skew is the most common pitfall

When a key dominates traffic (e.g., 30% of records), one partition becomes hot while others stay idle. Skew sources:

Hot entities (top orders, super users, popular products)

Enumerated keys with a small value domain (country, business line)

Time‑based keys (hour or minute timestamps)

Mitigation strategies:

Add a random suffix to hot keys (key + random bucket)

Use a composite key such as user_id:action_type Avoid time‑based keys altogether.

6.3 Partition count is not “the more the better”

Increasing partitions raises the concurrency ceiling but makes rebalancing expensive. When partitions are added, the hash routing changes, breaking existing order guarantees. Rebalance time grows linearly with partition count, and controller elections slow down. A practical rule of thumb is to provision partitions at 1.5× the expected peak throughput, assuming each partition can sustain 5‑10 MB/s.

7. Schema contract – topics are not raw byte pipes

7.1 Three typical accidents without a schema registry

Silent field rename: user_id → uid breaks 30 downstream consumers.

Silent field deletion: a producer removes a field that a data warehouse still depends on.

Semantic drift: status changes from numeric (1‑5) to an enum string, causing all consumers to misinterpret the value.

These failures happen because no gate prevents schema changes.

7.2 Schema Registry – turning contracts into enforceable rules

At ten‑million QPS the standard practice is to deploy a Schema Registry (Avro, Protobuf, or JSON Schema). Each topic registers a schema; producers attach the schema ID to the message header; consumers validate against the registered schema and reject mismatches. Schema evolution must obey compatibility rules (forward, backward, or full).

7.3 Compatibility tiers

Three levels of compatibility are illustrated (image omitted). The core rule for bidirectional compatibility is: add new fields with default values, never delete existing fields, and never change a field’s type.

8. Lifecycle governance – from creation to decommission

8.1 Creation workflow

Topic creation is no longer a one‑liner kafka-topics --create. A formal request form must include business domain, owner team, initial schema version, expected peak QPS and message size, retention policy, partition and replica count, and environment isolation level. After approval, an automated platform creates the topic and records metadata in a central store.

8.2 Evolution – change‑log for schema and partitions

Every schema change, partition expansion, or retention adjustment goes through a change‑process that records who initiated it, when it takes effect, affected producers/consumers, and a diff of before/after. This audit trail is essential for post‑mortems.

8.3 Decommission – “topic hospice”

When a topic’s traffic drops to zero, a multi‑step decommission process (illustrated in an image) ensures that owners, timelines, and downstream dependencies are verified before deletion. Skipping any step often leads to the classic accident “the topic had no traffic, I deleted it”.

8.4 Retention – cost and capacity driver

Retention policies have two dimensions: time (e.g., 7 days, 30 days) and size (e.g., 100 GB per partition). With tens of thousands of topics, retention directly impacts disk cost. Anti‑patterns include a blanket 7‑day retention for all topics and relying on default cluster settings without visibility.

9. Topic governance platform – from wiki to self‑service system

9.1 Core capabilities

When the topic count exceeds a thousand, a wiki or Excel sheet is insufficient. A governance platform should provide searchable registry, dependency graph, and integrated Schema Registry.

9.2 Dependency graph

The graph visualizes which producers write to a topic, which consumers read it, peak and average QPS, and the last active timestamp. It surfaces orphaned topics (e.g., 20% unused for six months) and stray producers.

9.3 Permissions and quotas

Default Kafka ACLs are coarse‑grained; at ten‑million QPS they must be refined to per‑topic write/read, schema‑change, and config‑change rights. Quotas prevent a single misbehaving producer from overwhelming the cluster.

10. From self‑discipline to mechanisms

Returning to the original test_v2_final topic, the incident happened because the organization lacked minimum standards for topic creation, ownership, and retirement. Introducing any of the mechanisms described—naming rules, schema registry, or dependency graph—would have prevented the failure.

The evolution of topic design mirrors a company’s message‑infrastructure maturity: ad‑hoc naming at 100 K QPS, informal conventions at 1 M QPS, and formal contracts, schemas, lifecycle, and governance at 10 M QPS. Skipping stages leads to instability.