Kafka Production Playbook: 6-Phase Deployment, Capacity Formulas & 10 Incident Runbooks
This article provides a comprehensive Kafka production deployment guide covering a six-phase rollout with mandatory checkpoints, three capacity planning formulas with examples, topic partitioning strategies, security and quota configurations, ten incident categories with symptom/mitigation/root-cause tables, major promotion assurance timeline, chaos engineering practices, and when to choose alternatives like RocketMQ or Pulsar.
Kafka Production Playbook: From Zero to Production
This article serves as the final chapter of a ten-part Kafka series, consolidating architecture, producer, consumer group, storage, replication, controller, performance, reliability, and monitoring into a complete production playbook. It answers three core questions: the correct sequence for production rollout (six phases with four checkpoints), the immediate actions to contain incidents (ten incident types with symptom/mitigation/root-cause separation), and the rhythm for major promotion risk elimination (pre-promotion timeline).
Three Main Threads
Six-phase rollout, each phase with a deliverable and four non-skippable checkpoints
Calculable capacity and topic planning methods — partition count is an irreversible decision
Ten incident categories in a three-column format: symptom → mitigation → root cause, with mitigation and root cause always executed separately
Three Real Incidents Illustrating Key Pitfalls
Capacity underestimated by 3x, disks full in 11 days : Only calculated daily message volume (2 TB) but forgot to multiply by replication factor (3), resulting in 6 TB/day actual growth. Emergency mitigation reduced retention from 7 to 2 days, but broke replay capabilities for reconciliation and debugging. Every term in the capacity formula has business meaning; missing replication factor removes entire replay ability.
Topic created with 1 partition, expansion broke order guarantees : Operations omitted --partitions, defaulting to 1 partition/1 replica. When traffic grew, emergency expansion to 24 partitions rehashed keys, scattering same-order messages across partitions and breaking downstream state machines relying on per-partition ordering for 6 hours. Partition count is one of Kafka's few irreversible decisions; better to leave 3x headroom than expand later.
No stress test, 8-minute retry storm : Pre-promotion estimated 3x traffic growth but only estimated, never stress-tested. At launch, request queues piled up, producer timeouts triggered automatic retries, retries amplified actual traffic 2.4x , creating a vicious cycle. P99 latency spiked from 9 ms to 5 seconds in 8 minutes. Mitigation: rate limiting + disabling non-critical topic writes. Root cause fix: stress-test calibration + retry backoff + quotas. Unverified capacity is just guesswork.
1. Technical Analysis: Zero-to-Production Playbook
1.1 Six Phases and Four Checkpoints
Divide Kafka cluster from machine procurement to traffic serving into six sequential phases:
Capacity Planning
Topic Planning
Cluster Deployment
Security & Quotas
Monitoring Integration
Chaos Drills
Each phase produces one deliverable. Order cannot be shuffled — monitoring before traffic is running blind; quotas before open access hands cluster fate to the wildest workload. Four checkpoints embedded in the flow: capacity table not signed → no topic creation; partition count not finalized → no writes; URP and lag metrics not live → no go-live; drills not passed → no traffic. Checkpoints exist not to slow rollout but to catch recoverable mistakes at the door — capacity errors are fixable, partition ordering corruption is a data accident.
The sequence has inherent logic; skipping steps causes hidden rework: capacity table feeds topic matrix (disk and partition counts determine parameters), topic matrix feeds deployment (storage layout planned by topic distribution), quotas feed monitoring (throttle alerts depend on quota existence), monitoring feeds drills (no metrics means no drill validation). Every step forward depends on the previous deliverable; time saved by skipping is repaid double as rework later.
1.2 Capacity Planning: Three Formulas and a Worked Example
Capacity planning needs no complex model, just three formulas:
Disk = Daily Increment × Replication Factor × Retention Days × 1.3 (1.3 = index + fluctuation buffer)
Partition Count = Target Throughput ÷ (Single-Partition Throughput × 0.7) (0.7 = safety watermark)
Broker Count = Target Throughput ÷ Single-Machine Bandwidth, then cross-check against total partitions
Take the larger of the two results — both disk and throughput constraints must be satisfied. Worked example: 2 billion 1 KB messages/day → 2 TB/day raw, 6 TB/day with 3× replication, 7-day retention plus buffer ≈ 55 TB; target peak throughput 800 MB/s → 120 partitions, 14 brokers. Two most common mistakes: forgetting replication factor (immediate 3× shortfall), using daily average instead of peak-day average — promotion-day traffic is 3–5× daily average, so average-based clusters get crushed on promotion day.
The matrix must also handle environment differences: same parameter matrix covers dev/staging/prod, with differences allowed only in retention and quota columns — if staging parameters drift too far from production, staging loses its "rehearsal" value. Parameter drift is the #1 cause of staging ineffectiveness : production parameter changes not synced back to matrix means rehearsals validate a non-existent configuration. Store matrix in repo, require code review for parameter changes, and environment consistency is guaranteed.
After calculation, one more step: pin budget numbers to monitoring dashboards . Actual disk watermark, actual throughput shown side-by-side with budget lines turns "when to expand" from gut feel into curve-watching — when daily watermark approaches budget line, expansion window is weeks away; waiting for disk alert means all-nighters.
1.3 Topic Planning: Partition Count Is an Irreversible Decision
Core of topic planning: partition count set once, set enough . Technically partitions can be expanded anytime, but with keyed writes, partition count change rehashes key mapping — "same order lands in same partition" local ordering assumption breaks instantly. Downstream consumers relying on local ordering (state machines, local caches, windowed aggregations) all require redesign. Partition count sizing: use formula from 1.2 for current need, multiply by 3× headroom, while respecting "max consumer concurrency" ceiling (partition count caps consumer parallelism). Naming convention equally critical: business-domain.data-type.environment (e.g., order.event.prod), paired with unified parameter matrix (Figure 3), turns new topic creation into form-filling instead of meetings.
Two counter-intuitive parameters in the matrix deserve elaboration. Behavioral logs: RF=2, min.insync=1 — this data is "losable but not blockable"; losing an hour of logs costs far less than ISR shrinking to 1 and rejecting all writes, so cost and availability trump completeness. CDC change streams: must use compact — downstream needs latest state per key via keyed lookup; delete policy destroys that capability. Parameter templates exist to make trade-offs explicit — every column is a deliberate choice, not a default.
1.4 Security & Quotas: Default Deny, Whitelist Allow
Multi-tenant clusters require security and quotas as infrastructure:
ACL default deny — each business only granted read/write on its own topics, preventing misconnections and privilege escalation
User-level quota — assign produce/fetch bandwidth ceiling per client, preventing a runaway workload (e.g., unbounded full replay) from consuming cluster bandwidth
Quota enforcement is soft: over-limit requests are delayed by broker rather than rejected — this turns "someone exceeded quota" into latency spikes instead of failures , and quota alerts (throttle time metric) become probes for detecting anomalous clients.
Quota parameter cheat sheet: producer_byte_rate / fetch_byte_rate limit per-user bandwidth; request_percentage limits request share. Clients can configure client.id prefix to share quota pool across all instances of same business. Three quota tiers: core business gets full quota with alert-only, ordinary business gets committed bandwidth, test/temporary tasks get low quota with special tag — when trouble hits, check throttle metric first; 80% chance it points directly to the over-quota temporary task.
1.5 Ten Incident Quick Reference: Mitigation and Root Cause Separated
Consolidates incidents from the nine prior articles into one table (Figure 4): ten high-frequency incident types, each with three fields — symptom (how detected), first action (how to stop bleeding), root cause fix (how to prevent recurrence). Single usage discipline: during mitigation only execute column two; after recovery return to column three . Mixing them is the main cause of secondary incidents — e.g., disk full: changing retention is mitigation, restarting broker turns mitigation into larger accident. Table value: at 3 AM no creative thinking needed — match symptom, execute column.
Beyond the ten, one "tail scenario" worth knowing boundaries: cluster healthy, all metrics green, but one business reports "messages slow". Usually not Kafka's fault — investigation order: client version/config changes → business processing latency → network path. Prove cluster innocent first, then investigate — basic skill for platform teams : compare same-period P99 of other businesses, one curve ends the argument.
1.6 Major Promotion Assurance: T-7 to T+1 Timeline
Promotion assurance essence: move all uncontrollable factors into controllable time windows .
T-7 Stress Test : replay production traffic or full-chain stress test to calibrate true cluster watermark, confirm capacity headroom ≥ 30% of estimated peak
T-3 Hardening : freeze non-essential changes, pre-expand capacity (pre-expanded machines must be pre-warmed — new brokers must have replicas migrated in advance, not start re-sync when traffic arrives)
T-1 Drill : walk through runbook, confirm on-call roster and escalation chain to named individuals
T0 Standby : core metrics on big screen, mitigation actions pre-authorized (rate limiting, degradation, write shutdown need no ad-hoc approval)
T+1 Retrospective : compare capacity budget vs actual curves, update capacity table
Runbook value is decided at T-7 — every action on promotion day should be a rehearsed action.
Retry storm called out separately as the classic promotion amplifier: broker slows → client timeouts → auto-retries → request volume doubles → broker slower. Three defense lines: client-side exponential backoff + jitter ( retry.backoff.ms with retry.backoff.max.ms to avoid synchronized retries); broker-side low quotas for non-core business to throttle them first; ops-side one-click degradation — disable non-critical topic writes to protect core path. All three lines must be in place by T-3; day-of is too late.
1.7 When Not to Use Kafka
Final chapter deserves fair talk: Kafka is not universal. Three scenarios seriously consider alternatives:
Complex routing semantics (content-based dynamic routing, delay queues, dead-letter retry topologies) — RocketMQ native support saves an order of magnitude engineering effort
Massive scale + want zero ops burden — Pulsar's tiered storage and compute-storage separation have structural advantages at 100k+ partition scale
Task queue semantics (per-message ack, priority, long-running tasks) — traditional MQ or Redis Streams fit better
Kafka's sweet spot: high throughput, ordered replay, multiple consumers independently consuming same data — streaming pipelines and event sourcing. Confirm your scenario sits in the sweet spot before selecting; forcing a non-sweet-spot scenario onto Kafka brings every pitfall early and hard .
2. Production Checklists & Runbooks
2.1 Config One: Full Production Broker Configuration
This config converges parameters from the entire series: reliability (articles 05/08), performance (07), controller (06), operations (09), ordered by rollout checklist. Every parameter annotated with what it protects — don't copy parameters you can't explain the protection for :
# ==== server.properties Production Full Skeleton ====
# --- Identity & Network ---
broker.id=1
# Rack awareness: cross-DC replica distribution depends on it
broker.rack=rack-1
listeners=PLAINTEXT://0.0.0.0:9092
# --- Reliability Trio (05/08) ---
default.replication.factor=3
min.insync.replicas=2
# Leader down: prefer unavailability over data loss
unclean.leader.election.enable=false
# --- Replica & ISR (05) ---
replica.lag.time.max.ms=30000
num.replica.fetchers=4
# Auto rebalance: new broker auto-migrates replicas
auto.leader.rebalance.enable=true
leader.imbalance.per.broker.percentage=10
# --- Performance (07) ---
num.network.threads=8
num.io.threads=16
# Flush to OS, reliability via replicas
# Forced fsync is performance suicide
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
# --- Retention & Disk (09) ---
log.retention.hours=168
# Single disk 85% triggers alert, 92% triggers proactive migration
log.segment.bytes=1073741824
log.cleanup.policy=delete
# --- Controller (06, KRaft) ---
process.roles=broker,controller
controller.quorum.voters=1@k1:9093,
2@k2:9093,3@k3:9093
controller.listener.names=CONTROLLER
# --- Quota Defaults (this article) ---
quota.producer.default=52428800
quota.consumer.default=1048576002.2 Config Two: Topic Matrix Script & Quotas
Topic creation not hand-typed; script generates from parameter matrix — write Figure 3 matrix as config file, create, validate, output manifest in one step:
# ==== Topic Matrix Script (driven by topics.yml) ====
# topics.yml one topic per line:
# name, partitions, rf, min-isr, retention
BS=localhost:9092
while IFS=, read -r T P R M RET; do
kafka-topics.sh --bootstrap-server $BS \
--create --if-not-exists \
--topic "$T" \
--partitions "$P" \
--replication-factor "$R" \
--config min.insync.replicas="$M" \
--config retention.ms="$RET"
done < topics.yml
# Immediate validation: partition count & ISR status
kafka-topics.sh --bootstrap-server $BS \
--describe | grep -E "Topic:.*Partition"
# Quota for new business (50 MB/s write)
kafka-configs.sh --bootstrap-server $BS \
--alter --add-config \
'producer_byte_rate=52428800' \
--entity-type users --entity-name svc-pay2.3 Promotion Stress Test Script
Stress test is not run once for a throughput number; it is calibration : measure single-partition throughput, single-machine bandwidth, queue depth — three baselines that feed capacity formulas:
# ==== Promotion Stress Test Three Steps ====
# Tool: kafka-producer-perf-test
# Step 1: Single-partition throughput calibration
kafka-producer-perf-test.sh \
--topic perf-1p --num-records 5000000 \
--record-size 1024 --throughput -1 \
--producer-props bootstrap.servers=$BS \
acks=all linger.ms=20 batch.size=131072
# Output avg throughput = single-partition baseline
# Step 2: Target load injection (80% estimated peak)
# throughput=640 means 640 MB/s cap
kafka-producer-perf-test.sh \
--topic perf-load --num-records 200000000 \
--record-size 1024 --throughput 640000 \
--producer-props bootstrap.servers=$BS \
acks=all
# Step 3: Watch four numbers during test
# P99 / RequestQueueTimeMs / URP / lagThree stress-test disciplines: shadow isolation — test uses dedicated topics ( perf- prefix), never inject into production topics; data sanitization — when replaying production traffic, sanitize first, set test topic retention to 1 hour to avoid disk explosion; results archived — each test's three baselines recorded in capacity table, next quarter's capacity planning references them directly, stress test compounds via calibrated baselines .
2.4 Retrospective: Three Incidents Point to Same Word
Re-examining opening three incidents: capacity missed replication factor, topic created with 1 partition, promotion skipped stress test — root cause identical: missing checkpoints . Capacity table signature takes ten minutes, nobody signed; topic parameter matrix finalization takes half day, nobody reviewed; stress test scheduling takes two days, cut by "time pressure". Checkpoints aren't bureaucratic theater; they force "mistakes everyone makes" through a mandatory sieve. Post-retrospective the team did three things: capacity table requires dual signature (business + platform), topics only creatable via matrix script (hand-typed commands need exception), promotion stress test written into schedule template (cut stress test = cut go-live date). Two years later, zero recurrence of same-class incidents. That is true production practice: incidents aren't scary; the same incident happening twice is scary.
2.5 On-Call Handbook One-Page Template
Last line of promotion defense is a one-page handbook, five sections:
What to watch (big-screen four quadrants mapping to Article 09's four perspectives)
What counts as anomaly (P99 > 3× baseline, URP non-zero, core topic lag time > SLA)
Mitigation actions (pre-authorized list + expected effect & side-effect per action)
Escalation path (on-call → platform lead → architect, every level's phone on paper)
Recovery actions (rate-limit removal condition: 15 consecutive minutes below baseline watermark)
Handbook printed at on-call station — screens crash, paper doesn't.
Rollout checklist's "tenth item" often missed: documentation . Cluster topology (machines, racks, roles), capacity table, topic matrix, quota ledger, on-call handbook, drill records — six documents are the cluster's household register. New hire's ability to build complete mental model in one hour depends on doc quality; incident-time ability to confirm "what role is this machine" also depends on them. Docs evolve with changes (every change syncs update); stale docs are more dangerous than no docs.
2.6 Version Upgrade & Migration from Rollout Perspective
Post-go-live cluster faces two "change-type major events". Version upgrade: minor versions rolling per-broker (within protocol compatibility window), major versions run full regression in staging for one week first. KRaft migration (Article 06's four phases) equally applies this article's methodology — it's just a "change-type rollout": capacity unchanged but control plane needs drills, monitoring must cover quorum metrics, rollback window pre-defined. Rollout methodology's value is it's not single-use: every major change is a new rollout.
3. Core Source Code: Four Critical Paths
3.1 Write Rejection: Where min.insync.replicas Takes Effect
// Produce request reliability gate
def appendRecords(...): Unit = {
// Reject write directly if ISR count insufficient
if (inSyncReplicas.size < minRequiredIsr) {
// NotEnoughReplicas exception -> client
// Client receives explicit error, not silent degradation
throw new NotEnoughReplicasException
}
localLog.append(...)
// acks=all: request parked in Purgatory,
// waits for all ISR to catch up before responding (Article 05)
delayProduce.tryComplete()
}3.2 Partition Expansion: Why Keys Get Scrambled
// Partition selection: rehash after expansion
def partitionFor(key: Array[Byte],
numPartitions: Int): Int = {
if (key == null) {
// No key: sticky partition, ordering irrelevant
stickyPartitionCache.partition(...)
} else {
// Has key: murmur2(key) % partitionCount
// Partition count 1 -> 24 changes result for every
// key -> local ordering assumption breaks (Incident 2 root cause)
Utils.murmur2(key) % numPartitions
}
}3.3 Retry Storm: Client-Side Amplifier
// Sender: retry & backoff (storm source)
if (batch.hasExpired || batch.attempts > 0
&& canRetry(batch)) {
// Backoff: base + jitter
// Without jitter, timed-out requests
// retry in perfect unison -> avalanche
long backoff = retryBackoff(batch);
reenqueue(batch, backoff);
}
// Exponential backoff ceiling client-controlled:
// retry.backoff.ms 100 ->
// retry.backoff.max.ms 50003.4 Quota Throttling: Why Soft Limit Doesn't Error
// ClientQuotaManager: over-quota handling
def maybeThrottle(...): Unit = {
// Calculate delay needed (by bandwidth delta)
val delayMs = throttleTime(...)
if (delayMs > 0) {
// Don't reject, hang response for delayMs
// Client sees "slower" not "error"
// This is soft limit: rate-limit not circuit-break
delayQueue.add(
new ThrottlingResponse(delayMs, req))
// throttle time also reported,
// lets well-behaved clients self-throttle
}
}4. Seven Accounts
Capacity Account : Missing replication factor = disk budget 3× short, disks full in 11 days. Every formula term has business meaning; calculated numbers must be dual-signed — signature purpose isn't blame, it's making two people actually calculate.
Margin Account : 3× partition headroom costs near zero (partitions are metadata only), while expansion cost is local ordering break + full downstream regression. Irreversible decisions deserve generous margins.
Stress Test Account : One full-chain stress test ≈ 2 person-days; one untested promotion incident losses in millions. Stress test output isn't "it holds" but three baselines — capacity formula inputs entirely depend on them.
Amplification Account : Retry storm turns 3× traffic into 7× (retry + 2.4× factor). Backoff + jitter + quota three lines, cost a few config lines, gain "graceful degradation" under overload.
Mitigation Account : Mitigation action checklist (rate limit, degrade, disable writes) pre-authorized, compresses promotion incident response from "call lead for approval" 15 minutes to 1 minute. Runbook currency is minutes.
Quota Account : Without quotas, one runaway replay task eats 80% cluster bandwidth; with 50 MB/s cap, it only hurts itself. Quotas are "fault isolation bulkheads" — mandatory for multi-tenant, not optional.
Selection Account : Using Kafka outside sweet spot costs 3–5× engineering effort of sweet-spot usage. Two extra days evaluation may save two years of hole-digging — final advice of final chapter.
5. High-Frequency Interview Questions (with Answers)
20 Q&A pairs covering: standard rollout flow & non-skippable steps, capacity formulas & common mistakes, why partition count irreversible, multi-tenant security/quota design, promotion timeline & node activities, retry storm definition & prevention, incident quick-reference discipline, unsuitable Kafka scenarios, soft quota design rationale, new broker onboarding steps, rack awareness purpose, stress test baseline calibration, topic naming importance, CDC topic parameter differences, behavioral log RF=2/min.insync=1 rationale, pre-expansion pre-warm meaning & why not day-of, one-click degradation pre-authorized actions, chaos drill scenarios & acceptance criteria, log retention policy design, one-word methodology summary ("checkpoints").
6. One-Page Cheat Sheet (Full Series Condensed)
Architecture in one line : Producer by key into partition, leader writes, ISR replicates, consumer pulls — everything revolves around "log + replica + offset" three things.
Reliability in one line : No-loss via trio (acks=all + min.insync=2 + unclean=false), no-dupe via idempotence & transactions, end-to-end via business idempotence fallback.
Performance in one line : Sequential write + page cache + zero-copy is foundation, batching three knobs are producer lever, partition count derived from throughput not more-the-better.
Monitoring in one line : Four perspectives × four signals, lag = LSO − committed by time, alerts three tiers, mitigation pre-authorized.
Operations in one line : Partition count set once, capacity by peak, mitigation & root cause separate, promotion work all in T-7.
Selection in one line : Kafka sweet spot = high throughput ordered replay + multi-consumer independent consumption; outside sweet spot, check RocketMQ & Pulsar first, don't force.
7. Series Conclusion
Ten chapters complete. Kafka's full picture compresses into a three-layer map (Figure 5):
Principle Layer (Articles 01-04) answers "why so fast" — log, partition, zero-copy, batching
Reliability Layer (Articles 05-08) answers "why trust data here" — replica & ISR, controller & KRaft, performance-reliability balance, exactly-once
Operations Layer (Articles 09-10) answers "what when things break" — monitoring system & production playbook
Each layer has distinct failure modes: weak principles → anti-pattern code; weak reliability → data loss; weak operations → small faults become major incidents.
Final compression of what this series truly delivers: parameters expire, versions upgrade, but three things don't — the mindset of understanding what each mechanism protects, the habit of calibrating with formulas and stress tests instead of gut feel, the organizational method of hardening incident experience into checkpoints. These three apply beyond Kafka: move to RocketMQ, Pulsar, or any distributed system, same playbook works.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
