Mastering Service Degradation: Strategies to Keep Your System Available Under Load

What Is Service Degradation

Service degradation means disabling or simplifying less important features when a system is under heavy load, similar to a tourist site limiting non‑essential activities during peak holidays to keep operations safe and efficient.

Service Level Definition

SLA (Service Level Agreement) is the benchmark for judging whether a load test is abnormal. Monitoring SLA metrics of core services during testing provides a clear view of system health. An SLA typically guarantees a certain uptime, often expressed as "six nines" (99.9999%).

Six Nines Meaning

Six nines correspond to 99.9999% availability, which translates to about 31 seconds of downtime per year.

Degradation Handling

Fallback Data

Examples include returning a default page when a service fails, providing static data, or using cached values.

Default Value : Set a safe default that will not cause data errors, e.g., inventory = 0.

Static Value : When a page or API cannot return data, show a static fallback or a retry prompt.

Cache : Use stale cache data if the fresh cache cannot be updated.

Rate‑Limiting Degradation

Rate limiting sets a maximum QPS for each request type; requests exceeding the threshold are rejected, protecting core services during traffic spikes. Friendly messages can inform users to retry later.

Timeout Degradation

Set a timeout for remote calls; if a call exceeds the limit and the feature is non‑critical, degrade it by hiding optional content (e.g., product recommendations) while keeping the main functionality intact.

Fault Degradation If a remote service fails (network, DNS, HTTP error), return default values, fallback data, static pages, or cached results. Retry / Automatic Handling Client‑side high availability can be achieved by providing multiple service endpoints. In micro‑services, mechanisms like Dubbo retries can be used. API retries should include a maximum retry count, a flag for service degradation, and asynchronous health checks. Degradation Switch When monitoring detects problems, a manual or automated switch can temporarily disable affected services. Switches can be stored in local config, Redis, Zookeeper, etc., and are also useful for gray‑release rollbacks. Crawlers and Bots Identify bot behavior (rapid actions, specific agents) and route them to static or cached pages. Read Degradation In a multi‑level cache hierarchy, if backend cache or DB is unavailable, serve data from front‑end cache or fallback data. Strategies include temporarily switching reads to cache/static content or blocking read access entirely. Write Degradation When write throughput exceeds DB capacity, temporarily write to an in‑memory store (e.g., Redis) and synchronize later, accepting eventual consistency. This applies to high‑traffic scenarios like flash sales or bulk reviews. Summary According to CAP and BASE principles, write operations prioritize availability over strict consistency. Degrading synchronous writes to asynchronous processes (cache, log, or batch updates) maintains high availability during spikes. Front‑End Degradation When the system is unstable, isolate requests as close to the user as possible, use local cache or fallback data, and, for low‑consistency scenarios (e.g., flash sales), provide mock data. JS Degradation Embed degradation switches in JavaScript to stop sending requests once thresholds are reached. Access‑Layer Degradation Use Nginx + Lua or HAProxy + Lua to filter invalid requests before they reach backend services, enabling early degradation. Application‑Layer Degradation Configure feature switches within the application to automatically or manually degrade services based on business conditions. In Spring Cloud, Hystrix provides circuit‑breaker and graceful degradation capabilities. Fragment Degradation When loading a page with many resources, missing some data can be compensated by substituting alternative content, ensuring the page remains functional. Pre‑Embedding Before major events (e.g., Double 11), static data can be pre‑loaded onto devices to reduce network load during the peak.