Microservices HA: Rate Limiting, Circuit Breaking, Degradation & Fallback with Sentinel

Why Microservices Inevitably Cascade into Failure

The greatest risk in microservices is not single-point failure but fault propagation . Service call chains form a mesh: gateway → application services → domain services → middleware → third-party APIs. When any node experiences timeout, latency, or errors, upstream requests block, threads are not released, and the system collapses.

Cascade Failure Chain: Thread pool exhaustion → new requests cannot be processed → service hangs → upstream retries bombard → more services block → full network avalanche .

Many teams wonder why systems run fine normally but collapse during promotions, peaks, or downstream jitter. The hard truth: your services lack fault tolerance and rely entirely on downstream stability.

Core principle of production-grade high availability: never trust your downstream. No matter how unstable downstream is, your own service must remain controllable, fallback-capable, non-crashing, non-exploding, and non-propagating.

Clearing the Confusion: Rate Limiting, Circuit Breaking, Degradation, Fallback

These four mechanisms solve fundamentally different problems; misusing them worsens failures.

🛡️ Rate Limiting: Traffic Protection (Prevent Self Overload)

Regardless of downstream health, intercept traffic exceeding service capacity.

Solves: traffic overload, burst peaks, malicious scraping, crawler bombardment.

⚡ Circuit Breaking: Fault Protection (Prevent Downstream Dragging You Down)

Detect downstream timeout/error rate spikes and trip the breaker to stop futile calls.

Solves: downstream jitter causing upstream thread pile-up and chain blockage.

🎯 Degradation: Avalanche Prevention (Actively Sacrifice Non-Core Capabilities)

Not passive failure but active strategy. During peak promotions, deliberately disable points, leaderboards, recommendations to protect ordering and payment core flows.

Solves: excessive traffic/pressure, actively protect core paths.

🔒 Fallback: Error Prevention (Guarantee User Experience Doesn't Hang)

When the chain fails, return fallback data, empty data, or cached data to avoid white screens and 500 errors.

Solves: system "available but not crashed", uninterrupted user experience.

One-sentence summary: Rate limiting guards the entrance, circuit breaking cuts faults, degradation preserves the core, fallback guards the experience.

Rate Limiting in Practice: Dual-Layer Architecture + Sentinel Implementation

Single-machine rate limiting is pseudo-limiting. Production standard requires gateway global rate limiting + service local rate limiting dual-layer defense.

1. Gateway Layer Rate Limiting (First Line of Defense)

Intercept malicious, oversized, illegal traffic before it enters business services.

IP rate limiting: block high-frequency requests from single IP, anti-crawler, anti-scraping.

User-dimension rate limiting: block high-frequency calls per account, anti-scalping.

Interface QPS rate limiting: global QPS quotas for core/high-risk interfaces.

Blacklist rate limiting: directly block malicious IPs/accounts.

2. Service Layer Rate Limiting (Second Line of Defense)

Prevent gateway failure, internal service calls, MQ burst traffic from crushing services.

Sliding window rate limiting: Sentinel default algorithm, smooth limiting, eliminates instantaneous traffic breakthrough, fits daily business traffic.

Token bucket rate limiting: suits burst peaks, allows instantaneous traffic pulses (preferred for promotions, flash sales).

3. Sentinel Rate Limiting Production Code (Ready for Deployment)

Core dependency for Spring Boot microservices:

① Annotation-based interface rate limiting (precise interface protection):

② Code flow control rule configuration (production-grade sliding window rate limiting):

4. Real Case: Flash Sale Crash Fix

🔥 Early Problem: Flash sale had no layered rate limiting; instantaneous 100k QPS directly crushed order service.

✅ Optimization: Gateway intercepts 80% invalid traffic + Sentinel token bucket smooth rate limiting + service layer QPS protection + queue mechanism to smooth traffic. Result: Service thread pool no longer explodes, system stably withstands super-large peaks.

Rate Limiting Core Principle: Better to rate-limit and error than to let service crash.

Circuit Breaking in Practice: Sentinel Rules + Code, Root-Cause Downstream Cascade Blocking

Biggest microservice hazard: weak dependencies kill main flow. Order service calls coupon, points, product detail. If points service jitters/times out/error rate soars, order service threads all block waiting, thread pool instantly piles up, triggering cascade failure.

1. Circuit Breaker Core Trigger Conditions (Production Standard)

Interface timeout rate exceeds threshold.

Exception error rate continuously spikes.

Slow call ratio too high.

2. Sentinel Circuit Breaker Three-State Principle

Closed State: Normal downstream calls, collect exception/timeout statistics.

Open State: Threshold triggered, direct short-circuit, no remote calls initiated.

Half-Open State: After breaker window ends, allow small traffic to probe downstream status, automatic self-healing.

3. Production-Grade Circuit Breaker Code Configuration (Exception Ratio Breaking)

4. Feign Remote Call Circuit Breaking + Fallback in Practice

⚠️ Production Circuit Breaking Pitfalls: ❌ Wrong: Unified breaker threshold for all interfaces. ✅ Correct: Differentiated thresholds for core vs non-core interfaces. ❌ Wrong: No fallback after breaking, directly throw exception. ✅ Correct: Breaker must pair with fallback strategy to ensure business availability.

Degradation Strategy in Practice: Lifesaver for High-Concurrency Promotions

Many teams confuse circuit breaking and degradation: circuit breaking is passive fault isolation, degradation is active core protection.

1. Degradation Classification

Manual Degradation: Manually disable non-core functions before promotions/peaks.

Automatic Degradation: Trigger degradation strategies automatically based on CPU, QPS, error rate.

2. Enterprise-Grade Degradation Trade-off Principles

✅ Priority Guarantee: Ordering, payment, fulfillment, refund core capital chains.

⚠️ Degradable Sacrifice: Points display, leaderboards, recommendations, history records, non-critical statistics.

💡 Weak Dependencies First: Peripheral capabilities not affecting main flow shut down first.

3. Real Business Case

E-commerce Promotion Peak: Actively degrade "user real-time points display", "product popularity leaderboard", "personalized recommendation". Fully release server resources to guarantee absolute stability of ordering and payment chains. Result: User experience slightly degraded, but core trading zero failures.

Fallback Strategy: System's Last Safety Bottom Line

After circuit breaking and degradation trigger, absolutely must not return 500 or throw exceptions directly. Fallback is the final defense for user experience and system stability.

Common Production Fallback Solutions

1. Cache Fallback: Downstream exception, return last successful cached data.

2. Default Value Fallback: Non-core fields return defaults, don't block main flow.

3. Empty Data Fallback: Query interfaces return empty list, no error.

4. Local Snapshot Fallback: Important configs, hot data fallback to local memory.

Fallback Core Idea: Allow data to be non-real-time, absolutely forbid business unavailability.

Four Mechanisms Linkage: Complete High-Availability Protection Closed Loop

Complete production protection chain:

Layer 1: Gateway Rate Limiting – Intercept illegal/oversized traffic, protect service entrance.

Layer 2: Service Rate Limiting – Secondary defense, prevent internal/MQ traffic breakthrough.

Layer 3: Circuit Breaking Cutoff – Downstream jitter/exception, break fault chain, avoid thread pile-up.

Layer 4: Active Degradation – Peak pressure too large, actively degrade non-core business, release resources.

Layer 5: Unified Fallback – All exception scenarios, unified fallback return, guarantee no crash, no explosion, no error.

This closed loop thoroughly solves: traffic peaks, downstream jitter, cascade faults, service avalanches.

High-Frequency Online Pitfall Review (Must-Read for Avoidance)

❌ Pitfall 1: Only rate limit gateway, not internal service traffic. Internal calls, MQ consumption can still crush service. ✅ Must have gateway + service dual-layer rate limiting. ❌ Pitfall 2: Unified breaker threshold globally, no business scene differentiation. Core business dragged down by non-core business breaking. ✅ Interface-granularity differentiated configuration. ❌ Pitfall 3: No fallback after breaking, directly throw exception. Failure directly exposed to users, experience crashes. ✅ Breaker must carry fallback strategy. ❌ Pitfall 4: Confusing circuit breaking with degradation. Strategy chaos, peak prevention makes it crash more. ✅ Faults use circuit breaking, peaks use degradation. ❌ Pitfall 5: No automatic recovery mechanism. Downstream recovers but business long unavailable. ✅ Half-open probing, automatic self-healing.

Full Article Core Summary (Including Code Implementation Points)

Microservice avalanche root cause is not bugs, but no fault isolation, no traffic protection, no chain fault tolerance ; high-availability protection is mandatory infrastructure.

Rate limiting, circuit breaking, degradation, fallback each have distinct roles: rate limiting prevents traffic, circuit breaking prevents faults, degradation preserves core, fallback preserves experience ; four mechanisms link to form closed-loop protection.

Production must implement dual-layer rate limiting : gateway global rate limiting + Sentinel service local rate limiting; prefer smooth queuing mode to eliminate instantaneous traffic breakthrough.

Circuit breaking is passive fault isolation (Sentinel exception/slow-call breaking), degradation is active architectural trade-off ; the two must never be confused.

All fault-tolerance strategies must pair with fallback methods: system rather degrade than crash, rather return fallback data than throw 500 exceptions .

Code implementation core: configure rules via Sentinel hard-coded configuration to avoid console config loss, meeting production high-availability stability requirements.