Second-Level Fault Detection at 10M QPS: Layered Signals & Safe Automation

At 1:47 AM, an order interface's success rate begins a slow decline. The monitoring platform samples once per minute, and alert rules require five consecutive points below threshold before firing. The first alert appears at 1:53 AM; the on-call engineer spends another three minutes confirming it isn't a reporting delay. By the time traffic is shifted, nearly ten minutes have passed since the first failed requests.

A postmortem often yields a single conclusion: "Change the collection interval from 60 seconds to 5 seconds." But after making that change, new problems emerge: metric cardinality explodes, transient spikes wake on-call engineers repeatedly, the monitoring backend buckles before the business services do, and alerts arrive faster but lack sufficient context for action.

Second-level detection aims to form a trustworthy judgment on high-risk faults before impact widens.

Why Minute-Level Detection Was Once Enough — and Why It Isn't Now

In a system handling a few thousand requests per second, six minutes might affect only tens of thousands of calls. Teams can rely on retries, manual confirmation, and post-hoc compensation to contain the damage. At millions or tens of millions of QPS, the same six minutes means a completely different exposure surface.

Assume a critical path carries 10 million QPS, and 1% of requests fail due to a dependency anomaly. That's 100,000 failures per second. Waiting six minutes to alert yields a theoretical 36 million accumulated failures. Real systems have degradation, retries, and traffic tiering, so the exact number differs, but the order of magnitude shows the problem: detection latency has become part of the impact scale.

Detection latency accumulates across multiple stages:

Each stage "only adds a little delay," but the total stretches silently. For example: signal generation waits 10 seconds, collection 15 seconds, aggregation 20 seconds, rule validation 30 seconds, notification another 10 seconds — totaling 85 seconds. Focusing only on collection interval makes it hard to find the slowest stage.

Therefore, don't rush to promise "5-second detection." First, break the target into a measurable budget:

This budget isn't a universal standard. Slow-growing disk capacity can tolerate minute-level checks; a core payment entry experiencing widespread timeouts may find even 10 seconds too slow. Speed must be dictated by the fault's propagation velocity.

Define "Detection" First — Otherwise Speed Is an Illusion

Many teams treat "a red dot on the monitoring dashboard" as detection, and "on-call engineer confirms the fault" also as detection. These two moments can be minutes apart, yet the metrics are mixed together, making it impossible to tell if optimizations actually work.

A more practical approach splits fault detection into four phases:

Signal Visible : The system produces a signal that reflects the anomaly.

Machine Judgment : Detection rules convert the signal into an anomaly event.

Impact Confirmed : Multiple pieces of evidence indicate users or key business flows are truly affected.

Action Triggered : The event reaches the on-call engineer or triggers an automated protection action.

Each phase must record its own timestamp. Only then can you distinguish whether signals are slow to arrive, algorithms are slow to judge, or notification paths are slow. You'll also discover that not every anomaly should escalate to "confirmed fault." Brief single-instance jitter can stay in a suspicious state, handled by local self-healing, without becoming a global event.

When measuring detection quality, at minimum track:

The metric to optimize is effective lead time; a fast-refreshing dashboard has little real value.

Fault Signals Must Evolve from "Machine Alive" to "Service Can Still Do Work"

Early fault detection often starts with heartbeats. A process reports "I'm alive" every few seconds; after several consecutive misses it's declared abnormal. It's simple, cheap, and good for detecting host crashes and process exits.

But a normal heartbeat only proves one probe path works. Thread pools may be exhausted, dependency calls may all time out, and error responses may be returned quickly. The process is alive, yet the service isn't completing the work users actually need.

Second-level detection requires layered signals, not pinning hopes on a single curve.

Layer 1: User Experience Signals

Synthetic probing closest to the user, client-side failure rates, and key page action success rates directly answer "Can the user still complete the task?" They are the most explanatory but not necessarily the best for localization.

For example, a payment button failure could originate from the gateway, order service, risk control, payment channel, or client network. User experience signals quickly confirm impact exists; subsequent layers narrow the scope.

Layer 2: Business Result Signals

Order creation count, payment success count, message delivery completions, login success counts — these bridge technical metrics and real loss. They must account for diurnal cycles, promotional traffic, and regional differences; fixed thresholds won't work.

The advantage of business results is filtering out "technically abnormal but business normal" noise. The downside is inherent latency — e.g., settlement completion spans multiple steps — so leading indicators must be retained alongside.

Layer 3: Service Golden Signals

Latency, traffic, errors, and saturation remain the four universal signal groups at the service layer. For second-level judgment, prefer counter increments and histogram buckets to avoid re-processing massive raw requests at the central end.

An interface's average latency showing no obvious change doesn't mean no fault. If the 99.9th percentile jumps from 200 ms to 4 seconds, the average may barely budge. In high-traffic systems, the absolute request count represented by tail latency is already large.

Layer 4: Dependencies, Resources & Runtime

Connection pool waits, thread pool queues, GC pauses, file descriptors, disk latency, network retransmits, and downstream errors provide evidence for localization. This layer has many signals; collecting all at 1-second granularity centrally is impractical. A more feasible strategy: retain low-cost summaries in steady state, temporarily increase sampling rate when entering a suspicious state.

This state-driven granularity adjustment is more controllable than "all data at 1-second forever."

Second-Level Detection Relies on Combined Evidence, Not More Sensitive Single Thresholds

Making alert thresholds more sensitive does surface anomalies faster, but it also surfaces normal jitter faster — especially in low-traffic shards, newly scaled instances, and during release cutovers, where a single window easily distorts.

Fixed Thresholds Suit Hard Boundaries

CPU > 90%, disk free < 5%, error rate > 10% — these rules are intuitive and explainable. They fit safety boundaries and catastrophic anomalies, but not complex business baselines.

Fixed thresholds must pair with minimum sample sizes. A cold endpoint with 2 requests in 5 seconds, 1 failure = 50% error rate. Alerting directly makes noise grow with system granularity.

Relative Baselines Detect "Today Is Different"

The same business varies greatly between weekday morning peaks and midnight. Baseline detection compares changes against the same time window, same region, or similar instances. It catches "hasn't crossed absolute threshold but has clearly deviated from normal" anomalies.

Baselines aren't better when more complex. Releases, holidays, and major promotions change normal patterns; without change context, the model may treat growth itself as a fault.

Rate of Change Catches Rapid Deterioration

When error rate rises from 0.01% to 0.2%, the absolute value seems low but has grown 20x. Rate of change and first-order differences identify worsening trends earlier, suitable for the early spread phase of a fault.

Rate of change is sensitive to small bases, so minimum event counts or minimum traffic thresholds are still required.

Multi-Window Balances Speed and Stability

Short windows sense quickly; long windows confirm the anomaly isn't transient. Using SLO error budget burn rate as an example, observe both 1-minute and 5-minute windows simultaneously: short window triggers on extreme burn rate, longer window confirms sustained elevation.

Below is a set of strategy examples for discussion, not a universal parameter set to copy:

Fast windows fight for time; slow windows build trust. The two must not substitute for each other.

Move Detection Closer to the Fault

Traditional monitoring ships all data to a central system for unified computation. This model eases management, but at massive scale the central pipeline simultaneously bears network transport, aggregation, storage, and query pressure. During incidents, metric spikes can make the central system slower precisely when the business needs it most.

Second-level detection requires layered judgment.

The local layer handles high-frequency, low-semantics data. It can count errors, queue depths, and latency histograms every second, sending only aggregates or anomaly events upward. The regional layer decides whether it's a single-instance jitter, a single-rack failure, or an entire availability zone issue. The global layer correlates multiple regions, services, and business results into a single event.

This design yields three direct benefits:

Reduce central throughput : Raw high-frequency signals aggregate at the edge; the center receives only summaries and events.

Lower judgment latency : Local protection doesn't wait for cross-region round trips.

Preserve fault isolation : When central monitoring is anomalous, nodes and regions can still perform basic detection.

But local judgment cannot have unlimited power. An instance's view is narrow; it can mistake its own resource pressure for a global traffic anomaly. Typically, allow local execution of reversible, limited actions — e.g., briefly reject new requests, evict a single instance, or increase sampling rate — while cross-region traffic shifting and large-scale degradation still require higher-layer evidence.

Push and Pull Need Not Be Mutually Exclusive

Pull models control collection cadence and detect target disappearance; push models suit short-lived tasks and event-type signals. Second-level systems often mix both:

Whether push or pull, every message must carry event time, source identity, and sequence information. Using only receipt time conflates network congestion with simultaneous business anomalies and makes duplicate/out-of-order detection difficult.

Keep the Monitoring Pipeline Trustworthy During Faults

Many second-level solutions perform well in normal load tests but fail during real incidents. The reason is clear: faults simultaneously generate more error logs, more retries, more metric labels, and more queries. The monitoring system's load often peaks exactly when the business needs it most.

Control Cardinality — More Important Than Collection Interval

If a metric carries user IDs, request IDs, or unbounded error-text labels, time series count balloons. Reducing collection interval from 60 seconds to 5 seconds only exposes the problem sooner.

High-frequency detection metrics should use controlled dimensions: service, endpoint, status code category, region, datacenter, release version. Request-level context belongs in logs or traces, linked via exemplar IDs — not stuffed into metric labels.

Reserve Degraded Channels

The monitoring pipeline itself needs its own service levels and degradation strategies. When storage slows, prioritize core SLOs, error rates, and protection states; lower sampling for debugging metrics. When notification platforms fail, send events via backup channels. When the global correlation layer is unavailable, regional detection must still deliver high-priority events directly to on-call engineers.

Time Must Be Comparable

Under a second-level budget, tens of seconds of clock skew can completely invert causality. Nodes need reliable time synchronization; events must record both event time and processing time. Stream processors must explicitly handle late data — a region arriving 3 seconds late shouldn't split one real fault into multiple events.

Time sync anomalies themselves should become monitoring signals. Otherwise you see "database errors first, application timeouts later" when the true order is reversed.

Monitoring Must Be Observed Externally

The monitoring platform cannot only monitor itself. Use independent black-box probes to periodically send known events and verify they complete collection, judgment, and notification within budget. This is an end-to-end synthetic test for the fault detection pipeline.

Second-Level Doesn't Mean Alert on Every Spike

When data updates every 5 seconds, the naive approach is to judge every 5 seconds and send alerts. This pushes many transient jitters to humans. After continuous paging, even critical alerts get ignored.

A better approach: keep fast detection, but separate "detection events" from "notify people."

Normalize Events First, Then Decide Whether to Notify

After a raw rule fires, generate a structured event containing service, region, impact scope, start time, evidence, and suggested actions. The event layer deduplicates, merges, suppresses, and escalates identical faults.

For example, a single database shard failure may simultaneously trigger connection timeouts, interface error rates, queue backlogs, and business success rate drops. Four rules shouldn't send four notifications. The correlation layer merges them into one event based on service topology and temporal proximity, marking the database anomaly as a candidate root cause — not a definitive conclusion.

Treat Changes as First-Class Context

Releases, config changes, scaling, and traffic shifts are critical clues. Alert events should automatically attach recent changes, but don't mechanically label the latest change as root cause. Temporal proximity is correlation; version comparison, rollback experiments, or trace evidence are needed.

Attach a Small Evidence Package to Every Alert

A second-level alert that only says "error rate exceeded threshold" still forces the on-call engineer to spend minutes opening different platforms. Better events include:

First anomaly time and most recent anomaly time;

Affected services, endpoints, regions, and versions;

Current value, baseline value, sample size, and duration;

Related user experience and business results;

Recent releases, config changes, and capacity changes;

Representative traces, log samples, and dashboard links;

Automated actions already executed and their outcomes.

An alert that's one second faster is less valuable than evidence that saves one minute of searching. The best second-level detection shortens both detection and judgment time.

Design False Positives, False Negatives, and Automation Authority Together

Every detection system trades off false positives and false negatives. The difficulty in second-level systems is that decisions are faster, leaving less time for human review. A low-quality rule directly wired to automated traffic shifting can cause more damage than the original fault.

Permissions can be tiered by "evidence strength" and "action reversibility":

Automated actions need four guardrails:

Scope limitation : Act on only one instance, one shard, or a small traffic slice at a time.

Duration limitation : Actions auto-expire unless the higher control plane explicitly renews.

Effect feedback : Immediately verify whether error rate, latency, and capacity improve after execution.

Fast rollback : Every action has a clear rollback path and full audit trail.

Rule evaluation can't just check whether alerts fired; it must replay past real incidents and normal peaks. Offline replay answers: if the new rule had been live then, how much earlier would we have detected, how many extra false positives, would it have misfired during a release window?

At 10M QPS, the Fundamental Shift Is in Judgment Architecture, Not Threshold Values

From 100K QPS to 1M QPS to 10M QPS, fault detection doesn't scale by linearly adding monitoring machines.

100K QPS: Complete the Basic Loop First

At this stage, coverage of core paths, unified metric definitions, on-call rotation, and postmortem processes matter most. Centralized monitoring, 15–60 second collection intervals, and simple multi-condition rules usually solve most problems.

Teams shouldn't rush to complex models. First ensure every alert has an owner, a dashboard, a runbook, and that the detection pipeline itself has availability monitoring.

1M QPS: Start Handling Partitioning and Noise

As services, regions, and instance counts grow, a single global average masks local faults. Detection must tier by region, availability zone, version, and key tenants while controlling label cardinality.

This stage should introduce event normalization, alert merging, change correlation, and multi-window rules. High-risk paths can shrink to 5–15 second granularity; ordinary resource metrics keep slower cycles.

10M QPS: Make Detection and Protection a Distributed Control System

At 10M QPS, raw observation volume, fault propagation speed, and automation action scope all change. Detection must sit near the signal source; regional layers can judge independently; the global layer correlates and authorizes. Monitoring backend overload isolation, data prioritization, and backup notification channels are no longer nice-to-have.

The fundamental change brought by scale is that the detection system evolves from an "observation tool" into a real-time system that participates in traffic control.

This also explains why cost can't be calculated by storage volume alone. Shorter windows bring more compute, higher peak throughput, and stricter control-plane reliability. Sustainable solutions allocate budget first to core user paths and fast-spreading faults; ordinary metrics continue on slower cycles.

Start Rolling Out from One Critical Path

If you're ready to push fault detection from minutes to seconds, pick one high-business-value path with a clear impact chain as a pilot. Overhauling all monitoring at once is costly and makes it hard to see each adjustment's effect.

Step 1: Define Fault Models

List the 3–5 fault classes this path most needs early detection for — e.g., widespread timeouts, single-region unavailability, downstream error propagation, thread pool exhaustion, abnormal business results. For each, document affected objects, propagation speed, and desired lead time.

Step 2: Draw the Latency Budget

From fault injection to notification delivery, measure each segment: signal generation, collection, aggregation, judgment, notification. Don't substitute configured values for real measurements. A config saying "5-second collection" doesn't guarantee data is queryable 5 seconds later.

Step 3: Build Combined Evidence

For each fault class, select one fast signal, one impact signal, and one localization signal. Fast signal buys time; impact signal controls false positives; localization signal guides remediation.

Step 4: Alert First, Automate Later

Run new rules in shadow mode for a period — only record "would have triggered" events, no notifications, no actions. Evaluate precision, recall, and effective lead time using historical incidents, normal peaks, and release windows.

Once rules stabilize, connect low-risk actions first. Every action must have scope limits, timeouts, effect verification, and rollback. Expand automation authority only after the team accumulates sufficient evidence.

Step 5: Validate with Fault Injection

Waiting for real incidents to validate detection takes too long and is uncontrolled. Inject latency, errors, resource exhaustion, and dependency unavailability in test environments and small production canaries to verify the full pipeline.

Each drill must record at least:

Actual MTTD and its P95, P99;

Whether alerts preceded user complaints and severe impact;

Whether similar rules duplicate notifications;

Whether evidence packages suffice for first-step remediation;

Whether automated actions stayed within expected scope;

Whether the monitoring system stayed fresh under fault traffic;

Whether recovery judgment was premature, causing flapping.

Step 6: Treat Rules as Software

Alert rules have versions, dependencies, and behavior changes — they should go through review, test, canary, and rollback. Clean up rules when services decommission; adjust windows when SLOs change; the release platform must feed structured change info to the event system.

Maintain an "ID card" for each high-priority rule:

Rule count isn't maturity. Being able to explain why each rule exists, when it fires, and what happens after it fires — that's maturity.

From "Seeing Anomalies" to "Preemptively Controlling Impact"

Moving fault detection from minutes to seconds appears to be a time reduction, but underneath it pulls signal design, computation placement, event models, monitoring reliability, and automation boundaries. Only increasing sampling frequency yields denser curves at best. When fast signals, impact evidence, and safe actions form a closed loop, fault exposure time truly shrinks.

Different systems need not chase the same number. Slow-growing capacity risks can be handled hourly; single-instance jitter can self-heal locally; only core entry points with massive failures justify expensive second-level pipelines. Applying speed where faults spread fastest and user loss is greatest is usually more effective than "full-site 1-second collection."

When the next fault occurs, do you want the team to first receive a "metric exceeded threshold" notification, or first receive an event that has confirmed impact, carries context, and has already completed the first protection step? That difference is exactly what second-level fault detection ultimately solves.