Skip to content
Production API Overload: Emergency Response, Root Cause Analysis & Resilience Patterns

Production API Overload: Emergency Response, Root Cause Analysis & Resilience Patterns

A comprehensive guide to handling production API overload covering emergency mitigation (rate limiting, degradation, circuit breaking), root cause diagnosis via monitoring and tracing, long-term optimizations (caching, async, DB tuning), with practical Java code for Sentinel, Caffeine+Redis, CompletableFuture, and Bloom filters, plus interview-ready troubleshooting frameworks.

Rain's Path to Java Mastery
Rain's Path to Java Mastery
Rain's Path to Java Mastery

Overall Handling Flow

The article presents a three-phase approach to resolve production API overload: emergency mitigation within 5 minutes, root cause diagnosis in 15-30 minutes, and long-term remediation.

Phase 1: Emergency Mitigation (within 5 minutes)

Rate Limiting : Enable gateway-level rate limiting immediately using token bucket algorithm (supports burst traffic). Set conservative threshold (e.g., 80% of usual peak). Prioritize core interfaces; rate-limit non-core ones.

Service Degradation : Disable non-core features (recommendations, comments, personalization). Return fallback data (static pages, cached data). Disable write operations, keep reads only.

Circuit Breaking & Isolation : Circuit-break services with high timeout/error rates. Use thread pool isolation per business module to prevent one service from dragging down the whole cluster.

Rapid Scaling : Leverage cloud elastic scaling to add service instances quickly. Prioritize stateless app services. Scale databases and caches cautiously to avoid data inconsistency.

Traffic Splitting : Route traffic to standby clusters if multi-active architecture exists. Disable canary releases; route all traffic to stable version.

Phase 2: Root Cause Diagnosis (15-30 minutes)

After mitigation, immediately locate the root cause for subsequent optimization.

Investigation Dimensions

Application Layer : Key metrics: CPU, Memory, GC, Thread Count. Common issues: infinite loops, memory leaks, frequent Full GC, thread blocking.

Network Layer : Key metrics: Bandwidth, Connections, Response Time. Common issues: network congestion, TCP connection exhaustion, slow queries.

Data Layer : Key metrics: QPS, Connections, Slow SQL. Common issues: index failure, lock contention, connection pool exhaustion.

Cache Layer : Key metrics: Hit Rate, Memory Usage. Common issues: cache penetration, cache breakdown, cache avalanche.

Monitoring Dashboard : Start with overall metrics, then narrow scope.

Log Analysis : Focus on error logs and slow request logs.

Stress Test Reproduction : Simulate production traffic in test environment to reproduce issue.

Tool Assistance : Use Arthas, JProfiler for live diagnostics.

Phase 3: Long-term Remediation

1. Code-level Optimization

Optimize slow methods: async and parallelize time-consuming operations.

Reduce nested loops: lower time complexity.

Avoid duplicate calculations: cache results in local variables.

Optimize object creation: reduce unnecessary object allocation and GC pressure.

2. Architecture-level Optimization

Service Splitting : Split monolith into smaller services by business domain for isolation.

Read-Write Separation : Reads go to replicas, writes to primary.

Async Processing : Offload non-real-time ops (SMS, logging) to async.

Message Queues : Peak shaving and decoupling.

3. Data Layer Optimization

Index Optimization : Add proper indexes for frequent queries.

SQL Optimization : Avoid SELECT *, avoid large table JOINs, use pagination.

Sharding : Horizontal or vertical split of large tables.

Connection Pool Tuning : Set pool size appropriately to avoid exhaustion.

4. Cache Layer Optimization

Multi-level Cache : Combine local cache (Caffeine) with distributed cache (Redis).

Cache Warm-up : Preload hot data into cache.

Cache Degradation : Return fallback data on cache miss.

Prevent Penetration/Breakdown/Avalanche : Use Bloom filters, mutex locks, randomized TTL.

Core Code & Technical Highlights

1. Gateway Rate Limiting (Sentinel Dynamic)

Highlight : Dynamic threshold adjustment without restart; dual QPS and thread count limiting.

@Configuration
public class SentinelConfig {
    @PostConstruct
    public void initFlowRules() {
        List<FlowRule> rules = new ArrayList<>();
        FlowRule productRule = new FlowRule();
        productRule.setResource("product:query");
        productRule.setGrade(RuleConstant.FLOW_GRADE_QPS);
        productRule.setCount(1000);
        productRule.setStrategy(RuleConstant.STRATEGY_DIRECT);
        productRule.setControlBehavior(RuleConstant.CONTROL_BEHAVIOR_RATE_LIMITER);
        productRule.setMaxQueueingTimeMs(500);
        rules.add(productRule);
        FlowRuleManager.loadRules(rules);
    }
    @Bean
    public BlockExceptionHandler blockExceptionHandler() {
        return (exchange, ex) -> {
            ServerHttpResponse response = exchange.getResponse();
            response.setStatusCode(HttpStatus.TOO_MANY_REQUESTS);
            response.getHeaders().add("Content-Type", "application/json;charset=UTF-8");
            String result = JSON.toJSONString(Result.fail("系统繁忙,请稍后再试"));
            DataBuffer buffer = response.bufferFactory().wrap(result.getBytes(StandardCharsets.UTF_8));
            return response.writeWith(Mono.just(buffer));
        };
    }
}

2. Multi-level Cache (Caffeine + Redis)

Highlight : Local cache first to reduce Redis network I/O; auto expiry and refresh; solves cache breakdown.

@Component
public class ProductCacheService {
    private final LoadingCache<Long, Product> localCache = Caffeine.newBuilder()
        .maximumSize(10000)
        .expireAfterWrite(5, TimeUnit.MINUTES)
        .refreshAfterWrite(1, TimeUnit.MINUTES)
        .build(this::loadProductFromDb);
    @Autowired private RedisTemplate<String, Object> redisTemplate;
    @Autowired private ProductMapper productMapper;
    public Product getProduct(Long productId) {
        try { return localCache.get(productId); }
        catch (Exception e) { log.error("本地缓存获取失败,productId:{}", productId, e); }
        String key = "product:" + productId;
        Product product = (Product) redisTemplate.opsForValue().get(key);
        if (product != null) return product;
        String lockKey = "lock:product:" + productId;
        Boolean locked = redisTemplate.opsForValue().setIfAbsent(lockKey, "1", 10, TimeUnit.SECONDS);
        if (Boolean.TRUE.equals(locked)) {
            try {
                product = (Product) redisTemplate.opsForValue().get(key);
                if (product != null) return product;
                product = productMapper.selectById(productId);
                if (product != null) {
                    redisTemplate.opsForValue().set(key, product, 30 + new Random().nextInt(10), TimeUnit.MINUTES);
                } else {
                    redisTemplate.opsForValue().set(key, new NullProduct(), 5, TimeUnit.MINUTES);
                }
                return product;
            } finally { redisTemplate.delete(lockKey); }
        } else {
            try { Thread.sleep(500); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
            return getProduct(productId);
        }
    }
    private Product loadProductFromDb(Long productId) {
        String key = "product:" + productId;
        Product product = (Product) redisTemplate.opsForValue().get(key);
        if (product != null) return product;
        product = productMapper.selectById(productId);
        if (product != null) {
            redisTemplate.opsForValue().set(key, product, 30 + new Random().nextInt(10), TimeUnit.MINUTES);
        }
        return product;
    }
}

3. Async Processing Optimization (CompletableFuture)

Highlight : Thread pool isolation per business; async callbacks and exception handling; improves throughput.

@Service
public class OrderService {
    private final ExecutorService orderExecutor = new ThreadPoolExecutor(
        Runtime.getRuntime().availableProcessors(),
        Runtime.getRuntime().availableProcessors() * 2,
        60L, TimeUnit.SECONDS,
        new LinkedBlockingQueue<>(1000),
        new ThreadFactoryBuilder().setNameFormat("order-thread-%d").setDaemon(true).build(),
        new ThreadPoolExecutor.CallerRunsPolicy()
    );
    @Autowired private SmsService smsService;
    @Autowired private LogService logService;
    public Order createOrder(OrderCreateDTO dto) {
        Order order = orderMapper.insert(dto);
        CompletableFuture.runAsync(() -> {
            smsService.sendOrderConfirmSms(order.getUserId(), order.getOrderNo());
        }, orderExecutor).exceptionally(e -> {
            log.error("发送订单确认短信失败,orderNo:{}", order.getOrderNo(), e);
            return null;
        });
        CompletableFuture.runAsync(() -> {
            logService.recordOrderLog(order.getOrderNo(), "CREATE", "订单创建成功");
        }, orderExecutor).exceptionally(e -> {
            log.error("记录订单日志失败,orderNo:{}", order.getOrderNo(), e);
            return null;
        });
        return order;
    }
}

4. Bloom Filter for Cache Penetration Prevention

Highlight : Distributed Bloom filter on Redis; supports dynamic scaling; low false positive rate.

@Component
public class ProductBloomFilter {
    @Autowired private RedisTemplate<String, Object> redisTemplate;
    private static final String BLOOM_FILTER_KEY = "bloom:product";
    private static final int EXPECTED_INSERTIONS = 1000000;
    private static final double FALSE_POSITIVE_RATE = 0.01;
    @PostConstruct
    public void init() {
        int numBits = BloomFilter.optimalNumOfBits(EXPECTED_INSERTIONS, FALSE_POSITIVE_RATE);
        int numHashFunctions = BloomFilter.optimalNumOfHashFunctions(EXPECTED_INSERTIONS, numBits);
        redisTemplate.opsForHash().put(BLOOM_FILTER_KEY, "numBits", numBits);
        redisTemplate.opsForHash().put(BLOOM_FILTER_KEY, "numHashFunctions", numHashFunctions);
        List<Long> productIds = productMapper.selectAllProductIds();
        for (Long productId : productIds) add(productId);
        log.info("布隆过滤器初始化完成,共加载{}个商品ID", productIds.size());
    }
    public void add(Long productId) {
        int numBits = (int) redisTemplate.opsForHash().get(BLOOM_FILTER_KEY, "numBits");
        int numHashFunctions = (int) redisTemplate.opsForHash().get(BLOOM_FILTER_KEY, "numHashFunctions");
        long[] hashes = getHashes(productId.toString().getBytes(), numHashFunctions);
        for (long hash : hashes) {
            long index = Math.abs(hash % numBits);
            redisTemplate.opsForValue().setBit(BLOOM_FILTER_KEY + ":bits", index, true);
        }
    }
    public boolean mightContain(Long productId) {
        int numBits = (int) redisTemplate.opsForHash().get(BLOOM_FILTER_KEY, "numBits");
        int numHashFunctions = (int) redisTemplate.opsForHash().get(BLOOM_FILTER_KEY, "numHashFunctions");
        long[] hashes = getHashes(productId.toString().getBytes(), numHashFunctions);
        for (long hash : hashes) {
            long index = Math.abs(hash % numBits);
            if (!redisTemplate.opsForValue().getBit(BLOOM_FILTER_KEY + ":bits", index)) return false;
        }
        return true;
    }
    private long[] getHashes(byte[] bytes, int numHashFunctions) {
        long[] hashes = new long[numHashFunctions];
        long hash1 = MurmurHash.hash64(bytes, bytes.length, 0);
        long hash2 = MurmurHash.hash64(bytes, bytes.length, hash1);
        for (int i = 0; i < numHashFunctions; i++) hashes[i] = hash1 + i * hash2;
        return hashes;
    }
}

Technical Challenges & Solutions

Dynamic Rate Limit Threshold : Fixed thresholds can't adapt to traffic fluctuations; adjusting requires restart. Solution: 1. Config center (Nacos/Apollo) pushes rules dynamically. 2. Auto-adjust based on monitoring (e.g., lower threshold when CPU > 80%). 3. Fine-grained limiting by user/IP/interface.

Graceful Recovery After Circuit Break : Traffic surge after recovery causes re-break. Solution: 1. Half-open state gradually allows traffic. 2. Exponential backoff to increase pass ratio. 3. Combine with metrics; fully open only when service metrics normalize.

Cache-DB Consistency : Stale cache causes inconsistency; dual-write inconsistency. Solution: 1. Update DB first, then delete cache. 2. Delayed double-delete for eventual consistency. 3. Distributed transactions for strong consistency scenarios.

Cross-shard Transaction Consistency : Cross-database transactions can't guarantee ACID. Solution: 1. Avoid cross-shard transactions via proper sharding design. 2. Flexible transactions (Seata TCC). 3. Eventual consistency with message queues.

Full-chain Stress Testing with Real Traffic : Test env differs from prod; can't simulate real traffic patterns. Solution: 1. Record prod traffic with tools like GoReplay. 2. Replay in test env. 3. Gradually amplify traffic to test limits.

Rapid Production Issue Localization : Complex issues; scattered logs and metrics. Solution: 1. Full-chain tracing (SkyWalking, Zipkin). 2. Arthas for live diagnostics. 3. Unified log format and levels.

Database Performance Under High Load : DB becomes bottleneck; connection pool exhaustion; severe lock contention. Solution: 1. Read-write separation. 2. Sharding. 3. SQL and index optimization. 4. Proper connection pool sizing.

Prevention Measures

Regular Stress Testing : Periodic full-chain stress tests to discover bottlenecks early.

Comprehensive Monitoring & Alerting : Build complete observability to detect issues promptly.

Capacity Planning : Plan capacity ahead of business growth.

Incident Response Playbooks : Develop and regularly drill runbooks.

Interview Simulation

The article includes a detailed mock interview demonstrating the three-phase methodology.

Emergency Response "Three Axes" (within 1 minute)

Rate Limiting : At gateway (Nginx/Spring Cloud Gateway) configure token/leak bucket. Example: Nginx limit_req_zone to 500 QPS, return 429 with friendly message.

Degradation : Non-core paths (recommendations, points) fallback via Sentinel/Hystrix to cached/empty data, preserving resources for core flows (order, payment).

Emergency Scaling : In K8s, kubectl scale to increase replicas; on VMs, add machines horizontally. Watch CPU and RT drop.

Root Cause Diagnosis: "Parallel Four Views"

Monitoring Dashboard (Prometheus+Grafana) : QPS, RT, error rate, CPU, memory, thread pool. QPS stable but RT spikes → internal logic slow; QPS multiplies → possible traffic spike or marketing event.

Slow SQL Panel (Druid/Arthas/SkyWalking) : Most load issues trace to a SQL suddenly doing full table scan. Check slow SQL top list.

Distributed Tracing (Jaeger/Zipkin) : See per-span latency: downstream user service taking 2s? Or local serialization eating CPU?

Thread Stack (jstack/Arthas thread -b) : If deadlock or threads blocked on a method, stack reveals the lock waited on.

Example: DB connections maxed out; slow SQL top was product query. Tracing showed massive cache invalidation → all requests hit DB → full scan on hundreds of thousands rows. Located in <3 minutes.

Long-term Fixes by Root Cause Category

Slow SQL : Add indexes, covering indexes, split large transactions, read-write separation. Use explain to check execution plan.

Cache Breakdown/Avalanche : Bloom filter, mutex lock, never-expire + async refresh. Redis SETNX for mutex.

Heavy Interface Logic : Async (MQ), parallel calls ( CompletableFuture). All non-mandatory sync returns go async.

Traffic Surge : Peak shaving (MQ buffer), CDN, page staticization. Common in flash sales.

Slow Downstream Dependency : Proper timeout + circuit breaker + thread pool isolation. Sentinel thread pool isolation.

Frequent GC : Tune GC strategy, reduce object creation, off-heap memory. Consider CMS/G1 under high concurrency.

Concrete Java example: An interface loop-called user service for avatars; at 100k users RT jumped from 50ms to 3s. Optimization: CompletableFuture parallelized the loop, then pushed downstream to provide batch query API. RT dropped back to ~80ms.

Architectural Long-term Prevention (Four Pillars)

Regular Stress Testing : Before major promotions, full-chain stress test with JMeter/Locust to find bottlenecks early.

Elastic Scaling Policies : K8s HPA based on CPU/memory; add predictive scaling using historical traffic.

Rate Limiting at Edge : Block malicious requests, crawlers at CDN/gateway; don't wait until backend.

Observability Stack : Ensure logs, metrics, traces complete; target: any fault localized within 5 minutes.

Additional Code Scenarios

Scenario 1: Gateway Token Bucket (Nginx + Lua)

Highlight : Millisecond-level limiting at traffic entry; dynamic config.

# nginx.conf
http {
    limit_req_zone $binary_remote_addr zone=perip:10m rate=5r/s;
    limit_req_zone $server_name zone=perserver:10m rate=500r/s;
    server {
        location /api/core {
            limit_req zone=perip burst=10 nodelay;
            limit_req zone=perserver burst=50 nodelay;
            error_page 503 =429 /rate_limit.html;
            proxy_pass http://backend;
        }
    }
}

Token bucket allows burst while enforcing average rate.

Dual-layer: perip prevents single-user abuse; perserver protects overall site water level.

Returns 429 directly; frontend shows "activity too hot" reducing useless backend pile-up.

Scenario 2: Cache Breakdown — Mutex Lock Loading

Highlight : Under high concurrency, only one thread loads hot data; others wait for result, avoiding DB crash.

public String getHotData(String key) {
    String value = redis.get(key);
    if (value != null) return value;
    String lockKey = "lock:" + key;
    try {
        if (redis.setnx(lockKey, "1", 10, TimeUnit.SECONDS)) {
            value = redis.get(key);
            if (value != null) return value;
            value = db.query(key);
            redis.set(key, value, 30, TimeUnit.MINUTES);
        } else {
            Thread.sleep(50);
            return getHotData(key);
        }
    } finally {
        redis.del(lockKey);
    }
    return value;
}
SETNX

implements lightweight distributed lock, preventing mass DB penetration.

Double-checked locking (DCL) ensures lock holder doesn't reload data.

Lock timeout prevents deadlock; release via Lua script for atomicity (simplified here).

Production combines logical expiry + async refresh to truly withstand hot key bursts.

Scenario 3: Interface Logic Optimization — CompletableFuture Parallel Calls

Highlight : Serial downstream calls in loop → parallel + batch; RT down 95%.

// Before: serial, RT = N * single latency
for (Long userId : userIds) {
    UserProfile profile = userService.getProfile(userId);
    profiles.add(profile);
}
// After: CompletableFuture parallel + custom thread pool
public List<UserProfile> getProfiles(List<Long> userIds) {
    List<CompletableFuture<UserProfile>> futures = userIds.stream()
        .map(id -> CompletableFuture.supplyAsync(() -> userService.getProfile(id), executor))
        .collect(Collectors.toList());
    return futures.stream()
        .map(CompletableFuture::join)
        .collect(Collectors.toList());
}
// Even better: push downstream for batch API
public List<UserProfile> getProfilesBatch(List<Long> userIds) {
    return userService.batchGetProfiles(userIds);
}
CompletableFuture.supplyAsync

+ custom thread pool turns serial network wait into parallel; RT from O(N) to O(1).

Thread pool size = CPU cores * 2 + network I/O wait factor; avoids excessive threads.

Best: reduce call count at source by pushing downstream for batch API; network round-trips from N to 1.

Technical Challenges & Solutions Summary

Hot Cache Invalidation Under High Concurrency : Risk: Tens of thousands requests penetrate to DB, dragging it down. Solution: Mutex lock + logical expiry + async refresh. Tools/Code: Redis SETNX distributed lock, Caffeine local cache.

Balancing Rate Limit Granularity & User Experience : Risk: Too strict hurts legitimate users; too loose crashes system. Solution: Multi-level limiting (IP + global) + graceful degradation hints. Tools/Code: Nginx limit_req, Sentinel flow control + Fallback.

Burst Traffic Peak Shaving : Risk: Flash sale/promo traffic spikes exceed system capacity. Solution: Message queue buffering + frontend queuing. Tools/Code: RocketMQ/Kafka async ordering, Redis counter queuing.

Slow Downstream Drags Caller : Risk: One slow dependency exhausts caller thread pool, causing avalanche. Solution: Thread pool isolation + timeout circuit breaker. Tools/Code: Sentinel thread pool isolation, Hystrix fallback.

CPU Spikes Without Obvious Slow SQL : Risk: Massive reflection/serialization/regex in loops. Solution: Flame graph to locate hot methods + parallelize/optimize algorithm. Tools/Code: Arthas profiler, CompletableFuture optimization.

Rapid Bottleneck Localization : Risk: Many dependencies; fault point hard to find; long diagnosis time. Solution: Full-chain tracing + unified monitoring + slow SQL/GC logs. Tools/Code: SkyWalking, Prometheus, Druid monitoring.

Emergency Scaling Not Fast Enough : Risk: Traditional app scaling takes minutes; traffic already crushed service. Solution: Containerization + HPA + image pre-warm + elastic policies. Tools/Code: Kubernetes HPA, pre-scaling playbooks.

Recurring Incidents : Risk: Fixes not institutionalized; next event repeats same issue. Solution: Regular full-chain stress testing + capacity planning docs. Tools/Code: JMeter/Locust stress tests, capacity waterline alerts.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

monitoringKubernetesRediscachingCompletableFuturehigh concurrencySentineltroubleshootingCaffeineNginxInterview Preparationrate limitingbloom filtercircuit breakersystem resilienceasync processing
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.