Designing for Failure: How Streaming Control Systems Stay Resilient

What Is Failure‑Oriented Design?

Failure‑oriented design treats failure as a first‑class concern, deliberately planning for every possible error scenario during the system design phase so that recovery strategies are built in from the start.

Why Design for Failure?

Failures are ubiquitous: hardware faults, software bugs, configuration mistakes, system degradation, traffic spikes, external attacks, and dependency outages. Even a brief failure can render services unavailable, damage user experience, and harm a company’s reputation; severe failures can cause permanent data loss and business collapse, as illustrated by the post‑9/11 impact on companies housed in the World Trade Center.

How to Design for Failure

Different lifecycle stages require distinct rules:

Design stage: simplify architecture, make layers clear.

Release stage: adopt minimal‑change principle, use small, frequent releases.

Operations stage: perform regular stress testing and keep dependencies minimal.

Case Study: Playback Control System Architecture

The playback control platform is split into three layers: an external service layer (SDK, query, change, and filter services), a core service layer (task scheduling and database services), and a data storage layer (distributed cache, database, open search). It provides three core pipelines: read, write, and filter.

Key characteristics:

Read/write separation and primary/secondary pipeline isolation prevent a failure in one link from affecting the core.

Core read services are decoupled from the database, allowing unlimited scaling under load.

Any link failure leaves the remaining links operational.

Database services enforce capacity‑based rate limits for query and change APIs.

Database Unavailability Fallback

A global switch diverts both read and write paths to a cache when the database is down. Writes are queued in a message broker and replayed once the database recovers, ensuring production traffic continues without data loss.

In the read path, if the fallback switch is on, the system checks the cache for resource availability. If the cache lacks the entry, it consults a blacklist before returning a result, thus avoiding database dependence.

Cache Redundancy and Consistency Checks

High‑concurrency workloads make the database a bottleneck, so the system uses cache as a buffer. Two mechanisms address cache‑DB inconsistency:

Redundant cache updates: on data change, the service updates the cache synchronously and also sends an asynchronous message to a background updater, guaranteeing both freshness and high availability.

Consistency detection: after a 5‑second cache entry expires (using Guava cache), a listener compares the cached data with the database and reports any discrepancy, triggering cleanup.

Dynamic Switching Between Remote and Local Calls

The platform offers both RPC‑based remote calls and an SDK for near‑side invocation. Near‑side calls reduce latency and improve success rates, but to protect stability the system implements a traffic‑dynamic‑switching degradation strategy: when SDK traffic exceeds a threshold, excess requests are automatically rerouted to the central control system, preventing overload on client machines even under extreme QPS.

Conclusion

Excellent architects are often pessimists who anticipate failure. By embedding failure‑oriented design into every stage—considering diverse error scenarios, preparing fallback mechanisms, and validating them through testing—systems can continue operating gracefully when unexpected problems arise.