Custom Spring Cloud Gateway Filters for Rate Limiting, Circuit Breaking & Gray Release

Why Not Use Sentinel Directly

The official Spring Cloud Gateway + Sentinel combination has three production pain points:

Long rule configuration propagation chain: Sentinel Dashboard → Nacos/Apollo → Gateway pull, causing high latency and weak consistency.

Missing canary capability: Sentinel focuses on rate limiting and circuit breaking; canary releases require an additional flagship system.

Coarse rule granularity: Sentinel defaults to rate limiting by route ID or service name, unable to support differentiated rate limiting by user tags within the same route.

The custom plugin goal: unify rate limiting, circuit breaking, and canary capabilities into Gateway's filter chain, store rules in Redis for real-time propagation, and make dimensions programmable.

Overall Architecture Design

The request flows through a chain of filters in this order:

Request enters Gateway
  ↓
[GlobalFilter: RouteContextFilter]   ← Parses routing rules, canary tags
  ↓
[LocalFilter: RateLimitFilter]       ← Token bucket / sliding window rate limiting
  ↓
[LocalFilter: CircuitBreakerFilter]  ← Circuit breaker judgment
  ↓
[LocalFilter: GrayRoutingFilter]     ← Canary routing match
  ↓
Forward to downstream service

Three filters are independent, ordered via FilterOrder, sharing a Redis rule storage layer.

Rule Storage Layer: Redis + Pub/Sub

3.1 Rule Data Structures

Rate limit rules stored as Hash with rule ID as key:

rate_limit:rule:order_api
  → limit_type: "TOKEN_BUCKET"
  → qps: 1000
  → burst: 200
  → key_resolver: "userId"   // Rate limit dimension

rate_limit:rule:payment_api
  → limit_type: "SLIDING_WINDOW"
  → qps: 500
  → window_size: 1
  → key_resolver: "ip"

Canary rules:

gray:rule:product_detail
  → match_tag: "vip"
  → match_value: "true"
  → target_version: "v2"
  → weight: 100          // Percentage traffic
  → fallback_version: "v1"

3.2 Real-time Propagation: Redis Pub/Sub

Rule changes publish events; all Gateway instances subscribe:

@Component
public class RulePublishService {
  @Autowired
  private RedisTemplate<String, String> redisTemplate;

  public void publishRuleUpdate(String ruleType, String ruleId) {
    String channel = "gateway:rules:" + ruleType;
    redisTemplate.convertAndSend(channel, ruleId);
  }
}

@Component
public class RuleSubscribeHandler {
  private final Map<String, Map<String, Rule>> ruleCache = new ConcurrentHashMap<>();

  @PostConstruct
  public void subscribe() {
    for (String type : Arrays.asList("rate_limit", "circuit_breaker", "gray")) {
      String channel = "gateway:rules:" + type;
      redisTemplate.execute(new SessionCallback<Object>() {
        @Override
        public Object execute(RedisOperations operations) {
          operations.subscribe(new RuleMessageListener(type, ruleCache), channel.getBytes(StandardCharsets.UTF_8));
          return null;
        }
      }, true); // pipeline mode
    }
  }
}

public class RuleMessageListener implements MessageListener {
  private final String ruleType;
  private final Map<String, Map<String, Rule>> ruleCache;

  @Override
  public void onMessage(Message message, byte[] pattern) {
    String ruleId = new String(message.getBody());
    // Fetch latest rule from Redis and refresh local cache
    Rule newRule = fetchRuleFromRedis(ruleType, ruleId);
    ruleCache.get(ruleType).put(ruleId, newRule);
  }
}

Key design: rules cached in JVM local memory, reads hit local Map with zero network overhead, updates propagate via Pub/Sub with sub-second effect.

Rate Limit Filter: Token Bucket + Sliding Window Dual Engine

4.1 Token Bucket Engine (Handles Burst Traffic)

public class TokenBucketRateLimiter {
  private final double capacity;     // Bucket capacity
  private final double refillRate;   // Tokens refilled per second
  private volatile double tokens;
  private volatile long lastRefillTime;

  public TokenBucketRateLimiter(double capacity, double refillRate) {
    this.capacity = capacity;
    this.refillRate = refillRate;
    this.tokens = capacity;
    this.lastRefillTime = System.nanoTime();
  }

  public synchronized boolean tryAcquire(int requested) {
    refill();
    if (tokens >= requested) {
      tokens -= requested;
      return true;
    }
    return false;
  }

  private void refill() {
    long now = System.nanoTime();
    double elapsed = (now - lastRefillTime) / 1_000_000_000.0;
    tokens = Math.min(capacity, tokens + elapsed * refillRate);
    lastRefillTime = now;
  }
}

4.2 Sliding Window Engine (Precise QPS Control)

public class SlidingWindowRateLimiter {
  private final int windowSize;      // Window size (seconds)
  private final int maxRequests;     // Max requests per window
  private final Deque<Long> requestTimestamps = new ConcurrentLinkedDeque<>();

  public synchronized boolean tryAcquire() {
    long now = System.currentTimeMillis();
    long windowStart = now - windowSize * 1000L;

    // Remove records outside window
    while (!requestTimestamps.isEmpty() && requestTimestamps.peekFirst() < windowStart) {
      requestTimestamps.pollFirst();
    }

    if (requestTimestamps.size() < maxRequests) {
      requestTimestamps.addLast(now);
      return true;
    }
    return false;
  }
}

4.3 Rate Limit Key Resolver (Programmable Dimensions)

@FunctionalInterface
public interface RateLimitKeyResolver {
  String resolve(ServerWebExchange exchange);
}

// Rate limit by user ID
public class UserIdKeyResolver implements RateLimitKeyResolver {
  @Override
  public String resolve(ServerWebExchange exchange) {
    String userId = exchange.getRequest().getHeaders().getFirst("X-User-Id");
    return userId != null ? userId : exchange.getRequest().getRemoteAddress().getAddress().getHostAddress();
  }
}

// Rate limit by API path + tenant
public class TenantKeyResolver implements RateLimitKeyResolver {
  @Override
  public String resolve(ServerWebExchange exchange) {
    String path = exchange.getRequest().getURI().getPath();
    String tenant = exchange.getRequest().getHeaders().getFirst("X-Tenant-Id");
    return tenant + ":" + path;
  }
}

4.4 RateLimitFilter Core Implementation

@Component
public class RateLimitFilter implements GlobalFilter, Ordered {
  @Autowired
  private RuleSubscribeHandler ruleCache;

  // One limiter instance per rule ID
  private final Map<String, RateLimitKeyResolver> resolverMap = new ConcurrentHashMap<>();
  private final Map<String, RateLimiter> limiterMap = new ConcurrentHashMap<>();

  @Override
  public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
    String routeId = exchange.getAttribute("routeId").toString();
    String ruleId = ruleCache.getRuleIdByRoute(routeId);

    if (ruleId == null) {
      return chain.filter(exchange);
    }

    RateLimitRule rule = ruleCache.getRule(ruleId);
    RateLimiter limiter = limiterMap.computeIfAbsent(ruleId, k -> createLimiter(rule));
    RateLimitKeyResolver resolver = resolverMap.computeIfAbsent(ruleId, k -> createResolver(rule));

    String key = resolver.resolve(exchange);
    boolean allowed = limiter.tryAcquire(key);

    if (!allowed) {
      exchange.getResponse().setStatusCode(HttpStatus.TOO_MANY_REQUESTS);
      exchange.getResponse().getHeaders().set("X-RateLimit-Remaining", "0");
      return exchange.getResponse().setComplete();
    }

    exchange.getResponse().getHeaders().set("X-RateLimit-Remaining", String.valueOf(limiter.remaining(key)));
    return chain.filter(exchange);
  }

  @Override
  public int getOrder() {
    return -2;  // Execute before RouteFilter
  }
}

Circuit Breaker Filter: Sliding Window Counting + State Machine

5.1 Circuit Breaker State Machine

public enum CircuitState {
  CLOSED,   // Normal, requests pass
  OPEN,     // Open, fast fail
  HALF_OPEN // Half-open, probe requests
}

public class CircuitBreaker {
  private final int failureThreshold;   // Failure count threshold
  private final int successThreshold;   // Success count threshold in half-open
  private final long resetTimeout;      // Wait time from OPEN to HALF_OPEN

  private CircuitState state = CircuitState.CLOSED;
  private int failureCount = 0;
  private int successCount = 0;
  private long lastFailureTime = 0;

  public synchronized boolean allowRequest() {
    switch (state) {
      case CLOSED:
        return true;
      case OPEN:
        if (System.currentTimeMillis() - lastFailureTime > resetTimeout) {
          state = CircuitState.HALF_OPEN;
          successCount = 0;
          return true;
        }
        return false;
      case HALF_OPEN:
        return true; // Allow but monitor
      default:
        return false;
    }
  }

  public synchronized void recordSuccess() {
    if (state == CircuitState.HALF_OPEN) {
      successCount++;
      if (successCount >= successThreshold) {
        state = CircuitState.CLOSED;
        failureCount = 0;
      }
    }
  }

  public synchronized void recordFailure() {
    failureCount++;
    lastFailureTime = System.currentTimeMillis();
    if (state != CircuitState.OPEN && failureCount >= failureThreshold) {
      state = CircuitState.OPEN;
    }
  }
}

5.2 CircuitBreakerFilter Implementation

@Component
public class CircuitBreakerFilter implements GlobalFilter, Ordered {
  private final Map<String, CircuitBreaker> breakerMap = new ConcurrentHashMap<>();

  @Override
  public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
    String routeId = exchange.getAttribute("routeId").toString();
    String ruleId = ruleCache.getRuleIdByRoute(routeId);

    if (ruleId == null) {
      return chain.filter(exchange);
    }

    CircuitBreaker breaker = breakerMap.computeIfAbsent(ruleId,
      k -> new CircuitBreaker(10, 5, 30000)); // 10 failures trip, 5 successes recover, 30s to half-open

    if (!breaker.allowRequest()) {
      exchange.getResponse().setStatusCode(HttpStatus.SERVICE_UNAVAILABLE);
      return exchange.getResponse().setComplete();
    }

    // Wrap downstream call, record success/failure
    return chain.filter(exchange)
      .doOnSuccess(v -> breaker.recordSuccess())
      .doOnError(e -> breaker.recordFailure());
  }

  @Override
  public int getOrder() {
    return -1; // After rate limiting, before canary
  }
}

Key detail: Circuit breaker is maintained independently per route, not a global singleton. Different downstream services' circuit breaker states do not affect each other.

Gray Release Filter: Tag Matching + Weighted Routing

6.1 Gray Routing Engine

public class GrayRoutingEngine {
  private final List<GrayRule> rules;

  public String routeTarget(ServerWebExchange exchange, String routeId) {
    String version = "v1"; // Default version

    for (GrayRule rule : rules) {
      if (rule.matches(exchange)) {
        // Hit canary rule, decide target by weight
        if (Math.random() * 100 < rule.getWeight()) {
          version = rule.getTargetVersion();
        }
        break; // Rules are mutually exclusive, stop at first match
      }
    }
    return version;
  }
}

public class GrayRule {
  private String matchTag;      // Tag name to match, e.g. "vip", "region"
  private String matchValue;    // Tag value to match, e.g. "true", "cn"
  private String targetVersion; // Target version
  private int weight;           // Canary traffic percentage
  private String fallbackVersion; // Fallback version if not matched

  public boolean matches(ServerWebExchange exchange) {
    String headerValue = exchange.getRequest().getHeaders().getFirst(matchTag);
    String paramValue = exchange.getRequest().getQueryParams().getFirst(matchTag);
    String actualValue = headerValue != null ? headerValue : paramValue;
    return matchValue.equals(actualValue);
  }
}

6.2 GrayRoutingFilter Implementation

@Component
public class GrayRoutingFilter implements GlobalFilter, Ordered {
  @Autowired
  private GrayRoutingEngine grayEngine;

  @Override
  public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
    String routeId = exchange.getAttribute("routeId").toString();
    String targetVersion = grayEngine.routeTarget(exchange, routeId);

    // Inject target version into request header, downstream routes by version
    ServerHttpRequest modifiedRequest = exchange.getRequest().mutate()
      .header("X-Target-Version", targetVersion)
      .build();

    return chain.filter(exchange.mutate().request(modifiedRequest).build());
  }

  @Override
  public int getOrder() {
    return 0; // Execute last, doesn't affect preceding rate limiting/circuit breaking
  }
}

Configurable Rule Management API

7.1 Rule CRUD REST API

@RestController
@RequestMapping("/admin/gateway/rules")
public class RuleAdminController {
  @Autowired
  private RulePublishService publishService;
  @Autowired
  private RedisTemplate<String, String> redisTemplate;

  @PostMapping("/rate-limit")
  public ResponseEntity<?> createRateLimitRule(@RequestBody RateLimitRuleDTO dto) {
    String ruleId = UUID.randomUUID().toString();
    String key = "rate_limit:rule:" + ruleId;
    redisTemplate.opsForHash().putAll(key, BeanUtils.beanToMap(dto));
    publishService.publishRuleUpdate("rate_limit", ruleId);
    return ResponseEntity.ok(ruleId);
  }

  @PostMapping("/gray")
  public ResponseEntity<?> createGrayRule(@RequestBody GrayRuleDTO dto) {
    String ruleId = UUID.randomUUID().toString();
    String key = "gray:rule:" + ruleId;
    redisTemplate.opsForHash().putAll(key, BeanUtils.beanToMap(dto));
    publishService.publishRuleUpdate("gray", ruleId);
    return ResponseEntity.ok(ruleId);
  }

  @DeleteMapping("/{type}/{ruleId}")
  public ResponseEntity<?> deleteRule(@PathVariable String type, @PathVariable String ruleId) {
    redisTemplate.delete("rate_limit:rule:" + ruleId);
    publishService.publishRuleUpdate(type, ruleId);
    return ResponseEntity.ok().build();
  }
}

7.2 Rule-to-Route Binding

@PostMapping("/admin/gateway/bindings")
public ResponseEntity<?> bindRuleToRoute(@RequestBody BindingDTO dto) {
  // routeId → ruleId mapping
  String key = "gateway:binding:" + dto.getRouteId();
  redisTemplate.opsForValue().set(key, dto.getRuleId());
  publishService.publishRuleUpdate("binding", dto.getRouteId());
  return ResponseEntity.ok().build();
}

Filter Execution Order & Thread Safety

8.1 Order Control

RateLimitFilter — Order: -2 — Purpose: Rate limit first, block excessive requests from hitting downstream

CircuitBreakerFilter — Order: -1 — Purpose: Circuit break next, protect requests that passed rate limiting from overwhelming faulty services

GrayRoutingFilter — Order: 0 — Purpose: Canary last, doesn't interfere with previous two interception layers

8.2 Thread Safety Points

Limiter Map uses ConcurrentHashMap, but each limiter instance internally requires synchronization (token bucket refill operation).

Circuit breakers isolated by routeId; different routes don't share state.

Rule cache uses volatile + ConcurrentHashMap dual guarantee to avoid visibility issues during Pub/Sub callbacks.

Don't store mutable state in exchange attributes inside filters; WebFlux is reactive non-blocking, exchange may be shared across threads.

Production Pitfalls & Solutions

Pitfall 1: Too Many Limiter Instances Cause OOM

Each routing rule created a TokenBucket instance; too many routes exhausted memory. Solution: Singleton limiters by ruleId using computeIfAbsent to ensure only one instance per rule.

Pitfall 2: Redis Pub/Sub Message Loss

Redis Pub/Sub doesn't guarantee persistence; rule changes during Gateway instance restart were lost. Solution: Use Redis Stream as persistent backup, pull full rules on startup.

// Pull historical rules from Stream on startup
redisTemplate.opsForStream().read(
  Consumer.from("gateway-group", "instance-1"),
  StreamReadOptions.empty().count(100),
  StreamOffset.create("gateway:rules", ReadOffset.lastConsumed())
);

Pitfall 3: Canary Weight Randomness Inconsistent in Distributed Environment

Different Gateway instances used different random seeds, same request could route to different versions. Solution: Use deterministic routing via request feature hash (e.g., userId hash) to guarantee same user always hits same version.

Pitfall 4: Circuit Breaker Success Count Not Triggered in Reactive Chain

WebFlux's doOnSuccess/doOnError work at signal level, but some swallowed exceptions wouldn't trigger. Solution: Use onErrorResume as unified fallback to ensure all failures are recorded to circuit breaker.

return chain.filter(exchange)
  .doOnSuccess(v -> breaker.recordSuccess())
  .onErrorResume(e -> {
    breaker.recordFailure();
    return Mono.error(e);
  });

Performance Benchmarks

Single Gateway instance (4C8G), load test results:

RateLimitFilter single processing latency : 0.02ms (local cache hit)

CircuitBreakerFilter single processing latency : 0.01ms (state machine judgment)

GrayRoutingFilter single processing latency : 0.03ms (tag matching + random)

Three filters combined total latency : < 0.1ms

Rule hot update latency (Pub/Sub) : < 50ms

QPS capacity (before rate limiting) : 12000+

With Redis rule cache hits, the three filters' overhead is negligible. The bottleneck lies in downstream services, not the Gateway layer.