Mastering Service Degradation: Strategies to Keep Systems Available

What Is Service Degradation

If you have read my previous analysis of service rate limiting, understanding service degradation becomes easy. Imagine a tourist site that normally allows free entry, but during peak holidays like Chinese New Year or National Day, visitor numbers surge and the site limits simultaneous entries—this is rate limiting. Service degradation means cutting less important features when traffic spikes, reallocating staff to critical areas.

In the internet, similar degradation measures are used. For example, during Double 11 sales, orders could be placed but not returned or modified, ensuring service availability by prioritizing core business and disabling non‑essential functions.

Service Level Definition

SLA (Service Level Agreement) is a key metric for judging whether a stress test is abnormal. Monitoring SLA indicators of core services during testing provides a clear view of system health.

SLA represents the guaranteed uptime agreed between provider and user, often targeting six nines (99.9999%).

Meaning of Six Nines

Six nines correspond to 99.9999% availability, meaning only about 31 seconds of downtime per year.

Degradation Handling

Fallback Data

Examples include returning default values, static data, or cached content when a page fails.

Default Values : Safe defaults that won’t cause data issues, e.g., inventory set to 0.

Static Values : Provide static data or error messages when an API cannot return data.

Cache : Use old cache when updates fail.

Rate‑Limiting Degradation

Set a maximum QPS threshold for request types; excess requests are rejected to protect core services, often with friendly messages like “system busy, please try later.”

Rate limiting requires load testing to determine system capacity and is a common stability measure.

Timeout Degradation

Define timeouts for data calls; if a non‑critical service exceeds the timeout, degrade it by hiding optional content such as recommendations or reviews, ensuring the main shopping flow remains unaffected.

Failure Degradation

If a remote service fails (network, DNS, HTTP errors), return default or fallback data, static pages, or cached responses.

Retry/Automatic Handling

Client high‑availability: provide multiple service endpoints.

Microservice retry: e.g., Dubbo retry mechanism.

API call retry: after max retries, mark service degraded and probe asynchronously.

Web side: add retry buttons or automatic retries for better user experience.

Automatic retries must set retry counts and ensure idempotent handling.

Degradation Switch

When monitoring detects service issues, manually toggle switches to disable affected services, possibly stored in local config, databases, Redis, or Zookeeper.

Switches also help roll back to previous versions during gray‑scale testing.

Bot and Crawler Handling

Analyze bot behavior (rapid actions, agents) and route crawlers to static or cached pages.

Read Degradation

In multi‑level cache architectures, if backend cache or DB is unavailable, use frontend cache or fallback data to improve user experience.

Typical strategies:

Temporary Read Switch : downgrade to read cache or static content.

Temporary Read Block : block read entry or specific read service.

Read flow: access‑layer cache → application‑layer local cache → distributed cache → RPC/DB.

When distributed cache or RPC/DB fails, automatic degradation avoids calls, suitable for scenarios with relaxed read consistency.

Page, fragment, and async request degradations protect core threads and maintain service continuity.

Static‑to‑dynamic and dynamic‑to‑static switches allow pre‑generated static pages to serve during high traffic, with fallback to dynamic rendering if static pages fail.

Write Degradation

When database write capacity is insufficient, temporarily use in‑memory stores like Redis for writes, then asynchronously sync to the database, accepting eventual consistency for high‑traffic scenarios such as flash sales.

Similarly, user reviews can be written asynchronously or rate‑limited during spikes.

Summary : In CAP and BASE contexts, write degradation sacrifices immediate consistency for high availability; asynchronous consumption, caching, or logging can bridge the gap.

Frontend Degradation

When backend services are unavailable, isolate requests near the user, use local cache or fallback data, and in low‑consistency scenarios (e.g., flash sales) provide mock data.

JS Degradation

Embed degradation switches in JavaScript to prevent requests when thresholds are exceeded.

Access‑Layer Degradation

Use Nginx+Lua or HAProxy+Lua to filter invalid requests before they reach services, enabling automatic or manual degradation.

Application‑Layer Degradation

Configure feature switches in the application; in Spring Cloud, Hystrix provides graceful degradation and circuit‑breaking to isolate failures.

Fragment Degradation

During page loads (e.g., Taobao homepage), if some resources fail, replace them with alternative data to keep the page functional.

Pre‑Embedding

Before major events like Double 11, pre‑download static data to users' devices to reduce network load during the event.

Source: https://www.cnblogs.com/Courage129/p/14427020.html