Four Production‑Grade Defenses Against Redis Cache Penetration in High‑Concurrency Microservices

1. The root cause is not a Redis miss but a non‑existent key

In high‑traffic e‑commerce scenarios, requests for IDs that are not yet stored in MySQL and have no cache entry trigger a full request chain: Client → Gateway → Service → Redis miss → MySQL miss → null response, which repeats for every request.

Client
 → Gateway
 → Product Service
 → Redis miss
 → MySQL miss
 → return null
 → next request repeat

Under concurrency this leads to a cascade: massive miss traffic, DB connection pool exhaustion, thread‑pool blockage, upstream retries, and eventual system‑wide outage.

2. Clarifying terminology

Cache penetration

Data does not exist anywhere; every request repeatedly falls back to the source.

不存在的数据，无法自然形成缓存闭环

Cache breakdown

A hot key expires at the same moment for many requests, causing a burst of DB reads.

存在的数据，在热点时刻失去了缓存保护

Cache avalanche

Many keys expire simultaneously or Redis becomes unavailable, forcing all traffic to the backend.

缓存层整体失效，后端承接全部压力

The mitigation focus differs for each: identify non‑existence, limit hot‑key concurrency, or control expiration rhythm.

3. Why “non‑existent” is a system‑wide risk

3.1 Cache mechanisms only help existing data

Typical cache code returns the DB result when redis.get(key) is null, but never records the null, so the next request repeats the miss.

Object value = redis.get(key);
if (value != null) {
    return value;
}
return db.query(key);

3.2 High concurrency amplifies resource consumption

Repeated misses contend for Redis connections, thread pools, DB connections, buffer pools, CPU, and downstream retry budgets, consuming the whole system capacity rather than business value.

3.3 “Non‑existent” may be temporary

Three categories exist: illegal requests, data that will appear later (e.g., pre‑launch items), and recently deleted items. Governance must consider lifecycle and consistency.

4. Four production‑grade defense lines (not a single trick)

4.1 Entry‑level governance

Validate parameters early; reject obviously illegal IDs before they reach Redis.

@Getter
@Setter
public class ProductQuery {
    @NotNull @Min(1) @Max(99_999_999)
    private Long productId;
    @NotBlank
    private String tenantId;
}

Controller should fail fast:

@RestController
@RequestMapping("/api/products")
@Validated
public class ProductController {
    private final ProductFacade productFacade;
    @GetMapping("/{productId}")
    public ProductDetailResponse detail(@PathVariable @Min(1) Long productId,
                                         @RequestHeader("X-Tenant-Id") @NotBlank String tenantId) {
        return productFacade.queryDetail(productId, tenantId);
    }
}

Gateway rate‑limits should target abnormal patterns (single IP, high 404/empty ratio, sudden spikes from a tenant or crawler) rather than only total QPS.

4.2 Bloom filter existence check

Bloom filters give a cheap “definitely not” or “maybe” answer, perfect for filtering massive random keys.

if (!bloomFilter.contains(productId)) {
    cacheNull(cacheKey);
    return Optional.empty();
}

Parameters: expected element count n, acceptable false‑positive rate p. Size m = -(n * ln p) / (ln 2)^2, hash count k = (m / n) * ln 2. Example: 10 M IDs, 0.1 % false‑positive → bitmap of tens of megabits.

Initialize with Redisson:

@Component
public class ProductBloomInitializer implements ApplicationRunner {
    private static final String BLOOM_NAME = "product:exists:bloom";
    private final RedissonClient redissonClient;
    private final ProductRepository productRepository;
    @Override
    public void run(ApplicationArguments args) {
        RBloomFilter<Long> bloom = redissonClient.getBloomFilter(BLOOM_NAME);
        bloom.tryInit(10_000_000L, 0.001D);
        long lastId = 0L;
        int batchSize = 2_000;
        while (true) {
            List<Long> batch = productRepository.scanIdsAfter(lastId, batchSize);
            if (batch.isEmpty()) break;
            for (Long id : batch) {
                bloom.add(id);
                lastId = id;
            }
        }
    }
}

Keep the filter fresh: listen to MySQL binlog → Canal/Debezium → Kafka → consumer that adds new IDs to the Bloom and deletes possible stale negative‑cache entries.

4.3 Negative cache (empty‑value cache)

Store a sentinel value (e.g., "__NULL__") for keys known to be absent, with a short, jittered TTL.

public final class CacheConstants {
    public static final String NULL_MARKER = "__NULL__";
    public static final Duration NULL_TTL = Duration.ofMinutes(3);
    public static final Duration DATA_TTL = Duration.ofMinutes(30);
}

Lookup logic:

String cacheValue = stringRedisTemplate.opsForValue().get(cacheKey);
if (CacheConstants.NULL_MARKER.equals(cacheValue)) {
    return Optional.empty();
}
if (cacheValue != null) {
    return Optional.of(deserialize(cacheValue));
}

TTL should be 1‑5 minutes with random jitter to avoid synchronized expiration.

private Duration negativeCacheTtlWithJitter() {
    long seconds = 120 + ThreadLocalRandom.current().nextLong(30, 120);
    return Duration.ofSeconds(seconds);
}

Risk: stale negative entries (dirty empty) when a new record appears. Mitigation: after successful DB insert, delete the negative cache and update the Bloom before the TTL expires.

4.4 Concurrency convergence on the source

The goal is to ensure only one request actually hits the DB for a missing key. Use single‑flight (local) or distributed locks (Redisson RLock).

@Service
public class ProductReadService {
    private static final String BLOOM_NAME = "product:exists:bloom";
    private static final String CACHE_KEY_PREFIX = "product:detail:";
    private static final String LOCK_KEY_PREFIX = "lock:product:detail:";
    private final StringRedisTemplate stringRedisTemplate;
    private final RedissonClient redissonClient;
    private final ProductRepository productRepository;
    private final ObjectMapper objectMapper;
    public Optional<ProductDetailDTO> queryProduct(Long productId, String tenantId) {
        String cacheKey = CACHE_KEY_PREFIX + tenantId + ":" + productId;
        String cached = stringRedisTemplate.opsForValue().get(cacheKey);
        if (CacheConstants.NULL_MARKER.equals(cached)) return Optional.empty();
        if (cached != null) return Optional.of(readJson(cached));
        RBloomFilter<Long> bloom = redissonClient.getBloomFilter(BLOOM_NAME);
        if (!bloom.contains(productId)) { cacheNull(cacheKey); return Optional.empty(); }
        RLock lock = redissonClient.getLock(LOCK_KEY_PREFIX + tenantId + ":" + productId);
        boolean locked = false;
        try {
            locked = lock.tryLock(200, 3000, TimeUnit.MILLISECONDS);
            if (!locked) throw new IllegalStateException("query product lock timeout");
            // double‑check after acquiring lock
            String secondCheck = stringRedisTemplate.opsForValue().get(cacheKey);
            if (CacheConstants.NULL_MARKER.equals(secondCheck)) return Optional.empty();
            if (secondCheck != null) return Optional.of(readJson(secondCheck));
            Optional<ProductEntity> entityOpt = productRepository.findOnlineByIdAndTenant(productId, tenantId);
            if (entityOpt.isEmpty()) { cacheNull(cacheKey); return Optional.empty(); }
            ProductDetailDTO dto = ProductDetailDTO.from(entityOpt.get());
            stringRedisTemplate.opsForValue().set(cacheKey, writeJson(dto),
                CacheConstants.DATA_TTL.plusSeconds(ThreadLocalRandom.current().nextLong(60, 300)));
            return Optional.of(dto);
        } catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
            throw new IllegalStateException("query interrupted", ex);
        } finally {
            if (locked && lock.isHeldByCurrentThread()) lock.unlock();
        }
    }
    private void cacheNull(String cacheKey) {
        stringRedisTemplate.opsForValue().set(cacheKey, CacheConstants.NULL_MARKER, negativeCacheTtlWithJitter());
    }
    private Duration negativeCacheTtlWithJitter() { /* same as above */ }
    private String buildCacheKey(Long productId, String tenantId) { return CACHE_KEY_PREFIX + tenantId + ":" + productId; }
    private ProductDetailDTO readJson(String v) { /* Jackson deserialization */ }
    private String writeJson(ProductDetailDTO d) { /* Jackson serialization */ }
}

Key points: check cache → Bloom → lock → double‑check cache → DB → write normal or negative cache. Lock granularity must be per business key; avoid global locks.

5. Monitoring, alerts, and metrics

Beyond Redis hit‑rate, collect:

cache_hit_count

cache_null_hit_count

cache_miss_count

bloom_reject_count

bloom_false_positive_count

db_not_found_count

lock_wait_ms

/

lock_timeout_count

illegal_param_reject_count

Derived ratios (negative‑cache hit rate, Bloom reject rate, DB miss rate, lock contention) should be charted and alarmed on sudden spikes.

6. Common pitfalls

Only negative cache without entry validation → Redis memory filled with empty keys.

Only Bloom filter without sync → newly created items are blocked.

Coarse‑grained locks → whole service serialised.

Long negative‑cache TTL → stale “not found” responses.

Missing observability → cannot tell whether problem is Bloom, lock, or upstream traffic.

Ignoring multi‑tenant or multi‑dimensional keys → cross‑tenant data leakage.

7. Evolution roadmap

Basic protection: parameter validation + negative cache + jittered TTL.

High‑concurrency protection: fine‑grained distributed lock, hotspot throttling, gateway rate‑limit.

Existence index: introduce Bloom filter, full‑load init, incremental sync, delete blacklist.

Systematic governance: rule grey‑release, full monitoring, auto‑circuit‑breaker, unified control plane, component reuse across services.

8. Production checklist

Parameter validation and source authentication implemented.

Gateway rules for abnormal source traffic configured.

Negative‑cache sentinel defined (not raw null object).

Negative‑cache TTL short with random jitter.

Active invalidation of negative cache after successful insert.

Bloom filter size and false‑positive rate evaluated.

Bloom full‑load and incremental sync mechanisms in place.

Deletion handling via blacklist or periodic rebuild.

Fine‑grained lock or single‑flight to converge DB reads.

Metrics for null‑hit rate, Bloom reject rate, DB miss rate, lock contention collected and alarmed.

Failure‑mode tests for Bloom delay, Redis outage, lock contention and corresponding degradation strategies.

When these items are satisfied, cache penetration stops being a midnight alarm and becomes a manageable part of the high‑concurrency read path.