Spring Cloud Microservices Practice #6: Sentinel for Service Fault Tolerance and Rate Limiting

1. Service Fault Tolerance: Why It Matters

When a downstream service fails, the failure can cascade: the calling service times out, consumes threads, exhausts thread pools, and eventually brings down the entire system. The snowball effect is caused by a single service becoming unavailable, request timeouts occupying threads, thread‑pool exhaustion, and overall system collapse.

Solution Comparison

Hystrix : isolation via thread pool or semaphore; circuit‑break based on failure ratio; no built‑in rate limiting; provides Hystrix Dashboard; project is discontinued.

Resilience4j : isolation via semaphore only; circuit‑break based on failure ratio; no built‑in rate limiting; no dashboard; project is active.

Sentinel : isolation via thread pool or semaphore; multiple circuit‑break strategies; supports rate limiting (including hot‑parameter limiting); powerful console; actively maintained.

Sentinel Core Advantages

Lightweight with good performance.

Rich rate‑limiting strategies.

Powerful management console.

Supports hot‑parameter limiting.

2. Sentinel Quick Start

1. Add Dependencies

<!-- Parent pom -->
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.alibaba.cloud</groupId>
      <artifactId>spring-cloud-alibaba-dependencies</artifactId>
      <version>2021.0.5.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<!-- Service pom -->
<dependency>
  <groupId>com.alibaba.cloud</groupId>
  <artifactId>spring-cloud-starter-alibaba-sentinel</artifactId>
</dependency>
<dependency>
  <groupId>com.alibaba.csp</groupId>
  <artifactId>sentinel-datasource-nacos</artifactId>
</dependency>

2. Install Sentinel Dashboard

# Download
wget https://github.com/alibaba/Sentinel/releases/download/1.8.6/sentinel-dashboard-1.8.6.jar

# Start
java -Dserver.port=8858 -Dcsp.sentinel.dashboard.server=localhost:8858 \
     -jar sentinel-dashboard-1.8.6.jar

# Access
http://localhost:8858
# Default credentials: sentinel/sentinel

3. Docker One‑Click Installation

version: '3'
services:
  sentinel:
    image: bladex/sentinel-dashboard:1.8.6
    container_name: sentinel
    ports:
      - "8858:8858"
    environment:
      - AUTH_USERNAME=sentinel
      - AUTH_PASSWORD=sentinel

4. Service Integration

# application.yml
spring:
  cloud:
    sentinel:
      transport:
        dashboard: localhost:8858
        port: 8719
        eager: true  # eager initialization to see data immediately

3. Traffic Control (Rate Limiting)

1. QPS‑Based Limiting

@RestController
@RequestMapping("/api/article")
public class ArticleController {

    @GetMapping("/list")
    @SentinelResource(value = "article:list", blockHandler = "handleBlock", blockHandlerClass = ArticleBlockHandler.class)
    public Result listArticles(@RequestParam Integer page) {
        return Result.success(articleService.list(page));
    }
}

public class ArticleBlockHandler {
    public static Result handleBlock(Integer page, BlockException ex) {
        return Result.error(429, "Too many requests, please try later");
    }
}

2. Concurrency‑Based Limiting

Configure in the console:

Resource: article:detail Threshold type: Concurrent threads

Single‑machine limit: 5

@GetMapping("/{id}")
@SentinelResource(value = "article:detail", blockHandler = "handleBlock")
public Result getArticle(@PathVariable Long id) throws InterruptedException {
    // Simulate slow query
    Thread.sleep(500);
    return Result.success(articleService.getById(id));
}

3. Associated Limiting

When interface A is limited, interface B is also affected. Example: limit /api/user/update when /api/user/login is being throttled.

Resource: /api/user/update
Flow control app: default
Threshold type: QPS
Threshold: 10
Associated resource: /api/user/login

4. Link‑Level Limiting

Define a flow rule that protects a common downstream method from being overwhelmed by multiple upstream services.

┌─────────┐
          │  Entry  │
          └────┬────┘
               │
        ┌──────┴──────┐
        │             │
        ▼             ▼
   ┌─────────┐   ┌─────────┐
   │ ServiceA│   │ ServiceB│
   └────┬────┘   └────┬────┘
         │             │
         └──────┬──────┘
                │
                ▼
          ┌─────────┐
          │ Common  │
          └─────────┘

# Disable URL aggregation to distinguish different links
spring:
  cloud:
    sentinel:
      web-context-unify: false

4. Circuit Breaking (Degrade)

1. Slow‑Call Ratio

# Console configuration
resource: article:detail
strategy: Slow call ratio
max RT: 500ms
ratio threshold: 0.5
min request count: 10
stat interval: 10000ms
degrade duration: 30000ms

Interpretation: if within 10 s there are more than 10 requests and 50 % exceed 500 ms, the resource is blocked for 30 s.

2. Exception Ratio

resource: user:update
strategy: Exception ratio
ratio threshold: 0.3
min request count: 10
stat interval: 10000ms
degrade duration: 20000ms

Interpretation: if within 10 s there are >10 requests and >30 % fail, block for 20 s.

3. Exception Count

resource: user:login
strategy: Exception count
exception count: 5
stat interval: 60000ms
degrade duration: 30000ms

Interpretation: if 5 exceptions occur within 1 min, block for 30 s.

4. Code Example

@Service
@Slf4j
public class UserService {

    @SentinelResource(value = "user:get", fallback = "getUserFallback", fallbackClass = UserFallback.class)
    public User getUser(Long id) {
        // Business logic that may throw exceptions
        return userMapper.selectById(id);
    }
}

public class UserFallback {
    public static User getUserFallback(Long id, Throwable ex) {
        log.error("Failed to get user: {}", ex.getMessage());
        return User.defaultUser(); // Return a default value
    }
}

5. Hot‑Parameter Limiting

1. What Is Hot‑Parameter Limiting?

Limit the request rate of a specific parameter, e.g., restrict the QPS of a particular userId:

GET /api/article?userId=123&page=1
# Requirement: limit the frequency of the same userId

2. Configure Hot‑Parameter Rule

@GetMapping("/list")
@SentinelResource(value = "article:list", blockHandler = "handleHotBlock")
public Result listArticles(@RequestParam Long userId, @RequestParam Integer page) {
    return Result.success(articleService.listByUser(userId, page));
}

# Console configuration
resource: article:list
param index: 0   # userId is the first parameter
threshold type: QPS
single‑machine threshold: 10
stat interval: 1000ms

3. Parameter Exception Items

# VIP users are exempt
param value: 1001
threshold type: QPS
single‑machine threshold: 100

# Regular users
default threshold: 10

6. System Adaptive Protection

1. System Load Protection

# Console configuration
- type: Load
  threshold: 3.0
- type: CPU usage
  threshold: 0.8
- type: Avg RT
  threshold: 500ms
- type: Entry QPS
  threshold: 1000

2. Code Configuration

@Configuration
public class SentinelSystemConfig {

    @PostConstruct
    public void initSystemRules() {
        List<SystemRule> rules = new ArrayList<>();
        SystemRule cpuRule = new SystemRule();
        cpuRule.setHighestSystemLoad(3.0);
        cpuRule.setHighestCpuUsage(0.8);
        SystemRule qpsRule = new SystemRule();
        qpsRule.setQps(2000);
        rules.add(cpuRule);
        rules.add(qpsRule);
        SystemRuleManager.loadRules(rules);
    }
}

7. Dynamic Rule Configuration (Nacos Persistence)

1. Why Dynamic Configuration?

Rules stored in the Sentinel console exist only in memory and are lost after a service restart.

2. Nacos Data Source Configuration

spring:
  cloud:
    sentinel:
      datasource:
        flow:
          nacos:
            server-addr: ${NACOS_HOST:localhost}:8848
            dataId: ${spring.application.name}-flow-rules
            groupId: SENTINEL_GROUP
            rule-type: flow
        degrade:
          nacos:
            server-addr: ${NACOS_HOST:localhost}:8848
            dataId: ${spring.application.name}-degrade-rules
            groupId: SENTINEL_GROUP
            rule-type: degrade
        system:
          nacos:
            server-addr: ${NACOS_HOST:localhost}:8848
            dataId: ${spring.application.name}-system-rules
            groupId: SENTINEL_GROUP
            rule-type: system

3. Example Flow Rule in Nacos

[
  {
    "resource": "user:get",
    "limitApp": "default",
    "grade": 1,
    "count": 100,
    "strategy": 0,
    "controlBehavior": 0,
    "warmUpPeriodSec": 10,
    "maxQueueingTimeMs": 500
  }
]

8. Common Issues and Pitfalls

Pitfall 1: Dashboard Shows No Data

Cause: Sentinel uses lazy initialization; data appears only after the first request.

spring:
  cloud:
    sentinel:
      eager: true  # eager initialization

Pitfall 2: Rate‑Limiting Rule Not Effective

Check the following:

Verify the resource name is correct.

Ensure the rule is not overridden by another rule.

Inspect Sentinel logs (e.g., logs/csp/sentinel-record.log).

Pitfall 3: Feign Integration Not Working

feign:
  sentinel:
    enabled: true

@FeignClient(name = "user-service", fallbackFactory = UserFallbackFactory.class)
public interface UserClient {
    // ...
}

9. Next Episode Preview

Spring Cloud Microservices Practice #7: SkyWalking Distributed Tracing

Principles of distributed tracing

SkyWalking deployment

Service integration

Performance analysis

10. Interactive Q&A

What problems have you encountered in service fault tolerance? How do you set rate‑limit thresholds? How do you recover after a circuit‑break? Share your questions in the comments; the author will help you analyze.