Skip to content
Kafka Monitoring & Operations: Building a Unified Metrics System from Lag to ISR
Operations 43 min read

Kafka Monitoring & Operations: Building a Unified Metrics System from Lag to ISR

This article details a comprehensive Kafka monitoring framework covering four-layer metrics, correct lag calculation (LSO minus committed), alerting tiers, operational SOPs, and practical configurations to reduce MTTD and alert noise.

Architecture Pony
Architecture Pony
Architecture Pony

1. Technical Analysis: Building the Monitoring System

1.1 Four-Layer Perspective: From Client to Consumer Group

Monitoring a Kafka cluster often fails by only watching brokers. The correct approach is layering: L1 Client SDK (send rate, request latency P99, error rate) shows user experience; L2 Broker (request queue wait, io-wait, network thread idle) shows machine capacity; L3 Replica & ISR (URP, offline partitions, ISR shrink rate) shows data safety; L4 Consumer Group (lag, commit rate, rebalance count) shows data flow. Each layer independently collects metrics and sets thresholds — any missing layer creates a blind spot where accidents hide.

Four-layer monitoring perspective: client / broker / replica / consumer group, each using golden signals
Four-layer monitoring perspective: client / broker / replica / consumer group, each using golden signals

1.2 Golden Four Signals: Latency, Traffic, Errors, Saturation

Layering solves "where to look"; four signals solve "what to look for". Applied to Kafka: Latency — P99 at every layer (produce request latency, queue wait, replica catch-up time, end-to-end lag time); Traffic — in/out bytes, message rates, consume rate; Errors — record-error-rate, request error codes, NotLeaderForFrequency; Saturation — queue length, disk usage, thread pool usage. Before adding a metric, ask which signal it belongs to; if you cannot answer, it usually doesn't need collection. This discipline cuts half the useless panels.

Common latency mistake: using average. Request latency is long-tailed; average normal but P99 exploding is typical — 1% of requests hit queue backlog, average barely moves, but user retries and timeouts all come from that 1%. All latency thresholds use P99 (or P99.9). Align client-side P99 with broker RequestQueueTimeMs: if client P99 high but broker queue low, time is spent in network or client, broker is innocent.

1.3 Correct Lag Algorithm: LSO − Committed

Lag definition seems simple but many calculate wrong. Accurate formula: lag = LSO - committed offset. LSO is "last stable offset" (marker of last completed transaction); without transactions it equals LEO. Common error: using "producer latest offset - consumer committed", which overestimates with transactions and misattributes across consumer groups. Three correct ways to view lag: per consumer group (different groups on same topic have unrelated progress), per partition (hot partition hidden by total), per time (count lag lacks business meaning; convert to time for SLA semantics).

Lag algorithm and three fake lags: multi-group sharing, transaction placeholders, offset resets
Lag algorithm and three fake lags: multi-group sharing, transaction placeholders, offset resets

Three "fake lags" called out: (1) Multi-group sharing topic: mixing committed offsets across groups makes group A's progress cover for group B; (2) In-flight transaction placeholders: uncommitted messages count in lag but read_committed consumers must wait — normal wait, not backlog; (3) Offset reset: manual reset makes lag instantly zero or spike, looking like auto-heal or explosion — actually a change operation. When lag curve jumps, check change logs first, then faults. Fake lags don't need fixing; need to avoid false alerts.

1.4 Time Lag Beats Count Lag

Count lag is incomparable: one partition at 5 msg/s with 1000 lag = 200 seconds; another at 5000 msg/s with 1000 lag = 0.2 seconds — same number, three orders of magnitude difference in business impact. Time lag (backlog messages / current consume rate) is the SLA language . Estimate: lag ÷ consume rate (5-min average) or use exporter's EstimatedTimeLag. Alert threshold: "lag time > 15 min" far more stable than "lag count > 1M" — former auto-adapts to traffic, latter floods false alerts on peak days. Trap: consume rate uses recent average; traffic drop (e.g., lunch break) inflates time lag, so threshold must allow traffic fluctuation margin.

1.5 Collection Pipeline & Cardinality Accounting

Five-stage pipeline: broker JMX → jmx_exporter (:9404) → Prometheus → Alertmanager → on-call. Most dangerous stage is cardinality. Partition-level metrics (one row per topic-partition) on 120k partitions = 3.6M series, single scrape takes 90s — exporter times out, Prometheus misses, broker wastes CPU recalculating. Solution: whitelist + pre-aggregation : only scrape MBean families needed for four layers (~19 families), series compressed to ~200; partition-level anomalies handled by on-demand query (daily inspection script full scan). Rule of thumb: 400-800 series per 10k partitions acceptable; beyond 50k must use Recording Rules.

Collection pipeline five stages and cardinality: full scrape 3.6M series vs whitelist ~200 series
Collection pipeline five stages and cardinality: full scrape 3.6M series vs whitelist ~200 series

Three collection disciplines: (1) Scrape timeout alerts separately: scrape failure (up==0) vs cluster slowdown are different; mixing them misjudges "monitoring down" as "cluster down" or vice versa. (2) Exporter as independent process: never embed in broker process; observation must not become failure source; give exporter its own resource limits to prevent large cluster dragging it down. (3) Critical metrics dual-channel: JMX plus kafka-exporter for consumer group lag; two channels cross-check — one down, other holds; metric discrepancies become clues.

1.6 Alert Tiering: P0 Has One Standard

Alert tiering protects on-call trust in the pager. Single gold standard: P0 = user already impacted (partition unreadable/unwritable, process gone, write rejected); P1 = user will be impacted soon (lag time exceeds SLA, disk near danger, ISR flapping); P2 = trend signal (rebalance high, P99 creeping up). Three disciplines: P0 must call phone (IM can be muted, phone cannot); every P1 must be acknowledged (ack action logged, auto-escalate if no ack); P2 never rings (only daily report, separating "watch trends" from "handle incidents" in time). If an alert fires 10 times in a month all false, fix or delete it — keeping it guarantees it will drown a real P0.

Alert three-tier matrix: P0 phone, P1 acknowledge, P2 daily report; three threshold methods combined
Alert three-tier matrix: P0 phone, P1 acknowledge, P2 daily report; three threshold methods combined

Threshold design: three methods stacked. Static thresholds (disk 85%, lag time 15 min) are fallback, simple but slow on gradual changes; Year-over-year (compare to same time last week, 3x higher) catches spikes, suits traffic metrics, auto-adapts to peak levels; Multi-metric correlation (lag up + commit stalled + CPU low) suppresses noise — single metric fluctuation 90% noise, three simultaneous degradations = real fault. Every alert annotation must state "what this means, first step to check" — alert copy is written for 3 AM bleary eyes.

1.7 Operational SOP: Sequence Matters More Than Restart

Operational danger lies in sequence and timing. Rolling restart five steps per broker: pre-check URP=0 (restart with out-of-sync replicas = self-inflicted unavailability) → stop one broker wait ISR converge → restart process wait replicas catch up → observe 5 min (URP / IsrShrinkRate / request P99) → next broker. Two pitfalls: controller node restarted last (avoid election stacked on replay); never move two brokers simultaneously — three-replica cluster with min.insync.replicas=2 loses writes if two replicas down. Disk waterline three tiers: 70% warning (archive + expansion ticket), 85% critical (stop non-critical writes, accelerate retention), 92% danger (actively migrate partitions to save cluster) — disk accidents are almost always gradual and predictable; waterlines turn accidents into planned expansions.

Operational SOP: rolling restart five steps per broker, disk three waterlines, change pre-check four questions
Operational SOP: rolling restart five steps per broker, disk three waterlines, change pre-check four questions

All changes (restart, partition expansion, migration, config change) must pass Change Pre-Check Four Questions : How many partitions affected (calculate exact number)? ISR will drop to what (below min.insync = outage)? Rollback time (path rehearsed)? Who watches dashboard (operator and observer separate)? If cannot answer, don't press enter. Sounds conservative, but prevents "confident senior engineer" errors: half of operational mistakes in postmortems were done by people who had performed the same operation dozens of times.

2. Production Practice: A Monitoring Config & One Postmortem

2.1 Config 1: jmx_exporter Whitelist & Prometheus Scrape

Config matches pipeline in Figure 3, single goal: keep series count low, capture all metrics needed by four layers . Partition-level dimensions entirely dropped; anomalies queried on-demand via inspection script.

# ==== jmx_exporter whitelist (kafka.yml) ====
# Principle: only scrape MBeans for four-layer perspective
# Partition-level dimensions (topic/partition) all discarded
startDelaySeconds: 0
lowercaseOutputName: true
rules:
# L3 Replica layer: URP & offline partitions
- pattern: 'kafka.server<type=ReplicaManager,
name=(UnderReplicatedPartitions|
OfflinePartitionsCount)><>Value'
name: kafka_replica_$1
type: GAUGE
# L2 Request layer: queue wait P99
- pattern: 'kafka.network<type=RequestMetrics,
request=(Produce|Fetch),
name=RequestQueueTimeMs><>99thPercentile'
name: kafka_queue_ms_$1
type: GAUGE
# L2 Request layer: end-to-end total time P99
- pattern: 'kafka.network<type=RequestMetrics,
request=(Produce|Fetch),
name=TotalTimeMs><>99thPercentile'
name: kafka_total_ms_$1
type: GAUGE
# L1 Throughput: in/out bytes & message rate
- pattern: 'kafka.server
<type=BrokerTopicMetrics,
name=(MessagesInPerSec|BytesInPerSec|
BytesOutPerSec)><>OneMinuteRate'
name: kafka_broker_$1
type: GAUGE
# L4 Consumer group lag: not in JMX,
# delegated to kafka-exporter (see below)
# ==== prometheus.yml scrape config ====
scrape_configs:
# broker JMX: 30s interval, 10s timeout
- job_name: kafka-broker
scrape_interval: 30s
scrape_timeout: 10s
static_configs:
- targets: ['k1:9404', 'k2:9404', 'k3:9404']
# Consumer group lag: 60s interval, cardinality controllable
- job_name: kafka-groups
scrape_interval: 60s
static_configs:
- targets: ['kafka-exporter:9308']
# Scrape health self-monitoring (monitoring the monitoring)
- job_name: kafka-exporter-self
static_configs:
- targets: ['kafka-exporter:9308']
# up{job="kafka-broker"}==0 alerts separately

2.2 Config 2: Full Alert Rules (P0/P1/P2 Tiered)

Rules correspond to Figure 3 three-tier matrix. Note each rule's for duration: too short triggers on spikes, too long misses window; P0 class 1-2 min, P1 class 5-15 min.

# ==== alerts-kafka.yml ====
groups:
- name: kafka-p0
rules:
# P0: partition unreadable/unwritable
- alert: KafkaOfflinePartitions
expr: kafka_replica_OfflinePartitionsCount
> 0
for: 1m
labels: {severity: P0}
annotations:
summary: "Partition unreadable/unwritable, call on-call"
first_step: "Check broker process and disk"
# P0: process gone
- alert: KafkaBrokerDown
expr: up{job="kafka-broker"} == 0
for: 2m
labels: {severity: P0}
# P1: lag time exceeds 15 min (SLA language)
- alert: KafkaLagTime
expr: kafka_group_lag_seconds > 900
for: 5m
labels: {severity: P1}
# P1: single partition lag > 3x group average
- alert: KafkaLagSkew
expr: kafka_partition_lag
> 3 * avg(kafka_partition_lag)
for: 15m
labels: {severity: P1}
# P1: disk available < 15%
- alert: KafkaDiskHigh
expr: node_fs_avail_ratio < 0.15
for: 10m
labels: {severity: P1}
# P2: rebalance > 5 per hour
- alert: KafkaRebalanceBusy
expr: increase(kafka_rebalance_total[1h])
> 5
labels: {severity: P2}

2.3 Inspection Script: Daily Ten-Item Checkup

Monitoring covers "real-time"; inspection covers "on-demand & full" — partition-level anomalies, slow replicas, log anomalies (high cardinality or low frequency) run daily via script, output only two states: all green, or list specific rows.

# ==== Daily inspection script (core items, excerpt) ====
BS=localhost:9092

# 1. Replica sync: output must be empty
kafka-topics.sh --bootstrap-server $BS \
--describe --under-replicated-partitions

# 2. Offline partitions: output must be empty
kafka-topics.sh --bootstrap-server $BS \
--describe --unavailable-partitions

# 3. Consumer group lag TOP10 (by time)
kafka-consumer-groups.sh --bootstrap-server \
$BS --all-groups --describe \
| sort -k6 -n -r | head -10

# 4. Disk waterline (>70% highlight)
df -h /var/kafka-logs | awk 'NR>1'

# 5. Quorum health (KRaft)
kafka-metadata-quorum.sh \
--bootstrap-server $BS describe

# 6. ISR shrink count (vs yesterday)
grep -c "Shrunk ISR" server.log.$(date +%F)

2.4 Postmortem: Single Partition Stuck for Two Hours

Replay accident one with this system: L4 layer has "single partition lag > 3x group average" LagSkew alert (P1, for 15m) — hot partition stuck would fire within 15 min; inspection script item 3 lists lag TOP10 daily at 9 AM, even if alert missed, next day appears in report; runbook says "single partition stuck → check that partition's consumer thread stack → common root causes: slow SQL, external call timeout". Three defense layers, any one present turns two hours into 15 minutes. Monitoring system value lies not in single-point accuracy, but in depth of layers.

2.5 Grafana Dashboard & On-Call Runbook

Dashboard organized by four layers into one-screen four quadrants: top-left L2 broker (request P99 & queue), top-right L3 replica (URP & ISR shrink), bottom-left L1 client (send rate & errors), bottom-right L4 consumer group (lag time TOP10 groups). Each chart answers one question; a panel that can answer "is it normal now?" within 10 seconds is a good panel . Runbook is one page: P0 fires → first action (check URP panel to scope impact) → decision trees for five common faults (process down / disk full / ISR drop / lag spike / quorum anomaly) → escalation paths. Runbook must update with every postmortem — postmortem without runbook update equals no postmortem .

3. Core Source Code: Where Metrics Come From

3.1 Lag Calculation: AdminClient Three Steps

// Lag calculation three steps: get committed, get end, subtract
Map<TopicPartition, OffsetAndMetadata>
committed = admin
.listConsumerGroupOffsets(group)
.partitionsToOffsetAndMetadata().get();

Map<TopicPartition, ListOffsetsResultInfo>
end = admin.listOffsets(specs)
.all().get();

// lag = LSO - committed
for (var e : committed.entrySet()) {
long lag = end.get(e.getKey()).offset()
- e.getValue().offset();
// With transactions end is LSO semantics,
// consistent with read_committed perception
report(e.getKey(), lag);
}

3.2 URP Statistic: Only Leader Has Vote

// ReplicaManager: URP statistic scope
def underReplicatedPartitionCount: Int =
// Only leader can judge if follower lagging
allPartitions.count { p =>
p.isLeader && p.isUnderReplicated
}

// Judgment basis is topic-level min.insync
def isUnderReplicated: Boolean =
inSyncReplicas.size < maxIsr

// Note difference between URP and ISR shrink:
// Shrink is event (rate), URP is state (level)

3.3 Disk Failure: Broker Doesn't Self-Heal, Controller Handles

// LogManager: single disk failure handling
def handleLogDirFailure(dir: String): Unit = {
// Step 1: find all partitions on that disk
val parts = logsByDir(dir).keys
// Step 2: mark local replicas offline
parts.foreach { tp =>
replicaState.markPartitionOffline(tp)
}
// Broker does NOT auto-exit: leader switch
// completed by Controller uniformly
// Recovery only via restart + full resync (throttled!)
}

3.4 Controlled Shutdown: Graceful Exit Before Restart

// Controlled shutdown: migrate leaders first then stop
def controlledShutdown(): Unit = {
// Notify Controller: I'm leaving
// Controller elects new leader per partition
while (!allLeadersMigrated) {
Thread.sleep(100) // wait migration complete
}
// vs kill -9:
// latter dumps all election pressure onto failover
// Rolling restart MUST take this path
}

4. Seven Accounts

Collection Account: Whitelist 200 series vs full 3.6M — four orders of magnitude difference. Full scrape not only wastes storage, but exporter and broker drag each other down: observation becoming failure source is monitoring's first commandment.
Discovery Account (MTTD): Accident one's two-hour discovery, replayed with defense-in-depth becomes 15 minutes. Every 10x MTTD reduction shrinks business loss more than 10x — loss accumulates over time, and later root cause harder to find.
Noise Account: One rolling restart 47 alerts, on-call muted and missed P0. Alert count inversely proportional to trust: weekly rings > 20, P0 inevitably drowned . Deleting false alerts as important as adding new ones.
Disk Account: 1.2TB resync saturated datacenter bandwidth. Set replication quota 50MB/s cap, 1.2TB resync takes 6.8 hours — looks slow but business P99 unaffected entire time, this account always pays off.
Window Account: Disk 70% to 97% under steady traffic usually days to weeks window. Waterline alerts turn "random emergency" into "planned expansion" — operational cost differs by order of magnitude.
Lag Semantics Account: Count lag 1M on 1KB messages may be 3 minutes; on 100KB messages may be 3 hours. Cross-cluster, cross-team SLA alignment only works with time lag; talking count lag is chicken-duck talk.
On-Call Account: Four-layer matrix + three-tier alerts + one-page runbook lets newcomers correctly respond to 80% of alerts. Monitoring system's ultimate output is not panels, but "anyone on-call will not sleep through a P0" certainty .

5. High-Frequency Interview Questions (with Answers)

Q1: Accurate lag formula? Common mistakes? Formula: lag = LSO - committed offset (per consumer group, per partition). Mistakes: (1) using producer latest offset instead of LSO, overestimates with transactions; (2) mixing committed across multiple consumer groups on same topic; (3) only watching total, hot partition averaged out; (4) only watching count, not time, detached from SLA.

Q2: Why time lag better for SLA? Count lacks rate info; same backlog count means vastly different business impact across traffic. Time lag = count / consume rate, directly maps to "how long until message processed", comparable across clusters and traffic. Rate uses recent average; traffic drop inflates time lag, threshold needs fluctuation margin.

Q3: Kafka monitoring four layers? Each solves what? L1 Client SDK (user feel: send rate, P99 latency, error rate); L2 Broker (machine capacity: queue wait, io-wait, network thread idle); L3 Replica & ISR (data safety: URP, offline partitions, ISR shrink rate); L4 Consumer Group (data flowing: lag, commit rate, rebalance count). Missing any layer = blind spot.

Q4: Golden four signals? Kafka mapping? Latency (each layer P99: produce latency, queue wait, replica catch-up, end-to-end lag time); Traffic (in/out bytes, message rates); Errors (record-error-rate, request error codes); Saturation (queue length, disk usage, thread pool usage). If cannot name signal when creating metric card, usually don't collect.

Q5: Metric cardinality explosion? Remediation? Partition-level metrics = partitions × metrics; 120k partitions × 30 metrics = 3.6M series → scrape timeout, storage bloat, broker recalc. Fix: whitelist only needed MBean families (~200 series), Recording Rules pre-aggregate, partition-level anomalies via inspection script on-demand full query. Rule: 400-800 series per 10k partitions.

Q6: Why jmx_exporter separate process? Embedded in broker, scrape storms (high cardinality, frequent) consume broker CPU/threads, observation becomes failure source. Isolated deployment isolates faults, allows separate resource limits and throttling; if exporter dies only "visibility" affected, not cluster.

Q7: Alert tiering standard? P0/P1/P2 response? Gold standard: "user impacted?" P0 = already impacted (partition unreadable, process gone, write rejected) → phone call must reach human; P1 = imminent impact (lag time > SLA, disk near danger) → IM acknowledge, timeout escalates; P2 = trend (rebalance high, P99 creeping) → daily report only, no ring. Discipline: weekly rings > 20 → clean false alerts, else P0 drowned.

Q8: Three threshold design methods? Scenarios? (1) Static: simple reliable, for hard physical boundaries (disk 85%, lag 15 min); (2) Year-over-year: compare same time last week, auto-adapts to traffic, for traffic metrics; (3) Multi-metric correlation: three metrics degrade together triggers, suppresses noise. Stack all three: static fallback, YoY catches spikes, correlation prevents noise.

Q9: Rolling restart correct sequence & checkpoints? Five steps per broker: confirm URP=0 → stop one wait ISR converge → restart wait replicas catch up → observe 5 min (URP/IsrShrinkRate/P99) → next. Key constraints: never two brokers simultaneously (3 replicas + min.insync=2 = write outage); controller last; restart via controlled shutdown (migrate leaders then stop).

Q10: Why controller last? Controller restart triggers leader election and metadata replay; all partition switches queue. If other brokers also restarting or catching up, failover pressure stacks, recovery significantly longer. Let normal brokers reach URP=0 stable first, then move controller, isolate election disturbance in clean environment.

Q11: Disk waterline three tiers? Actions? 70% warning: archive old data, file expansion ticket; 85% critical: stop non-critical topic writes, tighten retention to accelerate cleanup; 92% danger: actively migrate partitions to save cluster (migration with replication quota throttle). Disk accidents almost always gradual predictable; waterlines turn emergencies into planned expansions.

Q12: Single log dir failure handling? Why slow recovery? Broker doesn't auto-exit: partitions on that disk marked offline, leader switch by Controller, cluster continues serving (other replicas take over). Recovery slow: restart requires full resync, large disks take hours, must throttle replication quota else resync saturates network dragging normal business.

Q13: URP vs ISR shrink rate difference? URP (UnderReplicatedPartitions) is state: how many partitions out-of-sync now, level view, should be 0 long-term; ISR shrink rate is event: shrinks per unit time, trend view, normal near 0. High URP = "unsafe now"; frequent shrink = "replicas flapping in/out" (common causes: slow disk, network jitter, long GC). URP may intermittently zero, only watching state misses flapping faults.

Q14: Why lag curve jump check change logs first? Offset reset (manual skip/backlog replay), consumer group scale, topic partition expansion, upstream throttle all cause instant lag zero or spike, shape mimics fault. Check changes first to separate "human-caused waves" from "fault-caused waves", avoid debugging auto-heal illusion for two hours.

Q15: Frequent rebalance metric? Common root causes? Metric: rebalance count (hourly increase) and duration. Causes: consumer processing exceeds max.poll.interval.ms (kicked out), session timeout (network or GC), group membership dynamic scaling, heartbeat thread blocked. P2 monitor because not immediate fault but each rebalance pauses consumption, common source of lag spikes.

Q16: Distinguish scrape timeout (up==0) vs cluster slowdown? Separate alerts, separate panels. up==0 = exporter or network issue (observation down), cluster may be fine; cluster slowdown shows metric degradation (P99 up, queue growing). Trick: when scrape fails, CLI directly to broker verify (inspection script fallback); don't restart cluster blindly on "no metrics" — observation failure ≠ system failure.

Q17: Why lag alert use "single partition lag > 3x group average" relative rule? Total rule averages out hot partition fault (12 partitions, 1 stuck, total only +8%); absolute rule not reusable across traffic. Relative rule directly expresses "local anomaly": one partition significantly deviates from peers, regardless of traffic size worth suspicion. With for 15m filters spikes, false positive controllable.

Q18: Controlled shutdown vs kill difference? Controlled shutdown notifies Controller to migrate leaders per partition then stop — election pressure spread, clients barely notice; kill -9 makes all leaders vanish simultaneously, Controller handles hundreds of elections in failover path, recovery longer and ISR may shrink. Rolling restart MUST use controlled shutdown.

Q19: Recording Rules? What problem in Kafka monitoring? Prometheus pre-aggregation rules: high-frequency high-cardinality raw metrics (e.g., partition-level lag) aggregated by fixed dimensions (by group, by topic sum/max) into low-cardinality new metrics for dashboards and alerts. Effect: queries faster, dashboards don't explode, alert rules simpler; cost: lose partition detail — so partition anomalies handled by inspection script on-demand full query, two legs complementary.

Q20: Monitoring system "done" standard? Not panel count, three verifiable standards: (1) MTTD meets SLA — fault occurrence to awareness within SLA; (2) Alerts trustworthy — weekly ring count controlled, every alert acknowledged or cleaned; (3) Runbook effective — newcomer follows one-page runbook correctly handles 80% alerts, and runbook updates every postmortem. Monitoring is continuous operation, delivers "certainty" not panels.

6. One-Page Cheat Sheet

Four Layers: Client SDK / Broker / Replica & ISR / Consumer Group — missing one layer = one blind spot, blind spot location matches next accident.
Four Signals: Latency (P99 not average) / Traffic / Errors / Saturation — cannot name signal → don't collect.
Lag Iron Law: lag = LSO - committed; per group, per partition, per time; curve jump → check change logs first.
Cardinality Red Line: Partition-level metrics never scraped; whitelist + pre-aggregation; 400-800 series per 10k partitions, >50k must aggregate.
Alert Discipline: P0 phone, P1 acknowledge, P2 daily report; weekly rings > 20 clean false alerts first; alert copy written for 3 AM.
Ops SOP: Restart five steps per broker, controller last, never two at once; disk 70/85/92 three waterlines; change pre-check four questions unanswered → don't press enter.

7. Summary & Next Episode Preview

This episode in one sentence: Monitoring system = four layers × four signals (what to watch) + LSO-semantic lag (how to calculate) + three-tier alerts (what when rings) + SOP & inspection (what daily) . Four layers are spatial depth, three-tier alerts are response layering, inspection is temporal safety net — more depth, single-point failure less fatal. Relation to previous episodes: reliability design (episodes 05/08) decides "will fault lose data", monitoring ops decides "will fault become accident" — former by architecture, latter by discipline.

Next episode is Kafka series finale: Production Practice . First nine episodes disassembled parts; finale assembles whole machine: a complete zero-to-production launch checklist (capacity planning, topic planning, permissions & quotas, monitoring integration, chaos drills), plus quick-reference for ten high-frequency accident categories (each: symptom → locate → stop bleeding → root cure), stringing all series knowledge into "copy-to-work" battle handbook. Starting with a real cluster expansion.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

monitoringOperationsKafkaalertingPrometheusreliabilitySOPlagISRjmx_exporter
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.