Beyond Delayed Double Delete: How CDC Closed‑Loop Governance Solves Cache Consistency

Why the delete‑first vs. update‑first question is a pseudo‑problem

In a typical order‑payment flow the database is updated first ( status = PAID) and the Redis key is deleted. Because reads may hit a replica that has not yet received the transaction, a stale read can write the old value back to the cache, creating a dirty cache window (T1‑T7). The timeline is:

T1  支付回调到达，订单服务更新主库：status = PAID
T2  订单服务删除 Redis 中的 order:detail:1024
T3  用户立刻刷新订单详情，请求命中查询服务
T4  Redis 未命中，查询服务回源从库
T5  从库复制尚未完成，读到旧值：status = WAIT_PAY
T6  查询服务把旧值重新写回 Redis，TTL 30 分钟
T7  主从复制完成，但缓存已经重新变脏

Many teams try to fix the window with a “delayed double delete”:

updateOrderInDb(orderId, PAID);
redisTemplate.delete(cacheKey);
Thread.sleep(200);
redisTemplate.delete(cacheKey);

This works only if the sleep exceeds the actual replication delay. In high‑load or high‑latency environments (e.g., 800 ms lag) it fails, and the second delete may accidentally remove a newer value because of concurrent writes.

Binlog CDC as a reliable foundation

After a transaction commits MySQL writes a row‑level binlog. A CDC component (Canal, Debezium, etc.) captures the event, normalises it and publishes it to a message queue. Example payload:

{
  "eventId":"mysql-bin.003812:98423317",
  "table":"orders",
  "op":"UPDATE",
  "pk":"1024",
  "version":57,
  "updatedAt":"2026-07-16T01:42:18.317Z",
  "before":{"status":"WAIT_PAY","version":56},
  "after":{"status":"PAID","version":57,"user_id":9001,"merchant_id":3008}
}

Binlog guarantees that the event is emitted only after the transaction is durable, provides a stable offset for replay, and enables exactly‑once processing.

Production‑grade design: four planes

Control Plane : metadata that defines which tables/columns trigger which cache models, the mutation type (delete / upsert / rebuild), TTL, rate limits and feature switches.

Execution Plane : the consumer pipeline (Binlog → MQ → processor) that guarantees per‑key ordering, batch handling, back‑pressure and idempotent writes.

State Plane : observability of binlog positions, consumer offsets, lag, success/failure counts and dead‑letter queue size.

Governance Plane : runtime controls for enabling/disabling sync per table or tenant, replaying from a specific offset, manual key repair, and dynamic TTL/limit adjustments during traffic spikes.

Cache model strategies

Different cache views require different actions:

Detail caches (single key) – versioned upserts.

List caches – delete whole pages or use short TTL.

Set/collection caches – add/remove members.

Aggregated views – asynchronous recompute.

Use an explicit BIGINT version column instead of update_time because timestamps may be non‑monotonic across nodes.

Versioned cache example

{"version":57,"data":{"orderId":1024,"status":"PAID","amount":"199.00"}}

Consumers should apply a Lua script that writes only if the incoming version is newer:

local key = KEYS[1]
local payload = ARGV[1]
local newVersion = tonumber(ARGV[2])
local ttlMillis = tonumber(ARGV[3])
local currentVersion = redis.call('HGET', key, 'version')
if (not currentVersion) or (tonumber(currentVersion) < newVersion) then
  redis.call('HSET', key, 'data', payload, 'version', newVersion)
  redis.call('PEXPIRE', key, ttlMillis)
  return 1
end
return 0

Unified event model

public record ChangeEvent(
    String eventId,
    String table,
    Operation operation,
    String primaryKey,
    long version,
    Instant commitTime,
    Map<String, Object> before,
    Map<String, Object> after) {
  public enum Operation { INSERT, UPDATE, DELETE }
}

Cache projection interface

public interface CacheProjectionHandler {
  boolean supports(ChangeEvent event);
  List<CacheMutation> project(ChangeEvent event);
}

Example projection for order detail

@Component
public class OrderDetailProjectionHandler implements CacheProjectionHandler {
  private final OrderReadModelService orderReadModelService;
  public OrderDetailProjectionHandler(OrderReadModelService orderReadModelService) {
    this.orderReadModelService = orderReadModelService;
  }
  @Override
  public boolean supports(ChangeEvent event) {
    return "orders".equals(event.table());
  }
  @Override
  public List<CacheMutation> project(ChangeEvent event) {
    String orderId = event.primaryKey();
    String key = "order:detail:" + orderId;
    if (event.operation() == ChangeEvent.Operation.DELETE) {
      return List.of(new CacheMutation("order-detail", CacheMutation.MutationType.DELETE, key, event.version(), null, Duration.ZERO));
    }
    OrderDetailView detailView = orderReadModelService.loadFromPrimary(orderId);
    return List.of(new CacheMutation("order-detail", CacheMutation.MutationType.UPSERT, key, event.version(), Jsons.toJson(detailView), Duration.ofMinutes(30)));
  }
}

Cache mutation definition

public record CacheMutation(
    String cacheModel,
    MutationType type,
    String key,
    long version,
    String payload,
    Duration ttl) {
  public enum MutationType { DELETE, UPSERT, REBUILD }
}

Execution engine with idempotence

@Component
public class CacheMutationExecutor {
  private final StringRedisTemplate redisTemplate;
  private final ProcessedEventRepository processedEventRepository;
  private final RedisScript<Long> upsertScript;
  private final RedisScript<Long> deleteScript;

  @Transactional
  public void execute(ChangeEvent event, List<CacheMutation> mutations) {
    if (processedEventRepository.exists(event.eventId())) return;
    for (CacheMutation mutation : mutations) {
      if (mutation.type() == CacheMutation.MutationType.DELETE) {
        redisTemplate.execute(deleteScript, List.of(mutation.key()), String.valueOf(mutation.version()));
        continue;
      }
      if (mutation.type() == CacheMutation.MutationType.UPSERT) {
        redisTemplate.execute(upsertScript, List.of(mutation.key()), mutation.payload(), String.valueOf(mutation.version()), String.valueOf(mutation.ttl().toMillis()));
        continue;
      }
      throw new IllegalStateException("unsupported mutation type");
    }
    processedEventRepository.save(event.eventId(), event.commitTime());
  }
}

Kafka consumer with error classification

@Component
public class ChangeEventConsumer {
  private final ProjectionRouter projectionRouter;
  private final CacheMutationExecutor executor;

  @KafkaListener(topics = "${cache.cdc.topic}", containerFactory = "batchKafkaListenerContainerFactory")
  public void onMessage(List<ConsumerRecord<String, String>> records, Acknowledgment ack) {
    for (ConsumerRecord<String, String> record : records) {
      ChangeEvent event = deserialize(record.value());
      try {
        List<CacheMutation> mutations = projectionRouter.route(event);
        executor.execute(event, mutations);
      } catch (TransientRedisException ex) {
        throw ex; // retry by Kafka
      } catch (NonRetryableProjectionException ex) {
        sendToDlq(record, ex);
      } catch (Exception ex) {
        throw ex; // container‑level retry
      }
    }
    ack.acknowledge();
  }
}

The consumer distinguishes three failure categories: transient Redis errors (retry), non‑retryable projection errors (dead‑letter), and unexpected exceptions (container‑level retry).

Handling common edge cases

Duplicate messages : deduplicate by eventId and protect writes with version.

Out‑of‑order delivery : partition by primary key (e.g., orderId) to preserve ordering per entity.

Consumer backlog : batch processing, per‑key aggregation, back‑pressure and dynamic scaling.

Delete‑then‑repopulate race : versioned deletes ensure a stale delete cannot overwrite a newer value.

Read‑path consistency : critical keys read from the primary DB; non‑critical keys may read from replicas with version checks.

Governance features

Per‑table/tenant/model switches.

Time‑range replay using stored binlog positions.

Single‑key manual repair.

Dead‑letter queue with metadata (original event, failure reason, retry count, timestamps) and actions (auto‑replay, manual fix, ignore).

Dynamic TTL and rate‑limit adjustments for promotion periods.

Verification checklist before go‑live

Data correctness: ordering, idempotence, version protection.

Stability: Redis timeouts, consumer offset handling, CDC reconnection.

Observability: end‑to‑end lag, per‑model success rates, version drift.

Governance: runtime switches, replay capability, dead‑letter handling.

Conclusion

Cache consistency cannot be solved by fiddling with the delete‑update order. A reliable event‑driven pipeline that captures the true source of truth after the transaction commits, propagates it with per‑key ordering, version guarantees and built‑in replay/observability is required. When replication lag, hot‑spot churn, multi‑view dependencies or robust failure recovery appear, moving from “delayed double delete” to a Binlog‑based CDC solution with the four‑plane architecture provides a pragmatic, production‑grade consistency foundation.