How Midnight Key Expiration Spiked Redis CPU and the Optimizations That Fixed It

Background

During the early‑morning window the Redis 6.0 service repeatedly hit 100 % CPU for about one minute, causing higher latency for client requests and increased overall response time.

Analysis Process

Phenomenon Analysis

Monitoring showed a rise in master‑slave replication lag and a surge in DEL command executions, indicating that the master was deleting a large number of keys that expired simultaneously and propagating the deletions to the replica.

Root Cause Analysis

Code inspection revealed several modules that set cache keys to expire exactly at 00:00. When many such keys expire together two performance problems appear.

Redis Request Blocking

active_expire_effort defaults to 1 (range 1‑10); larger values consume more CPU.

Redis combines passive and active expiration. Passive expiration has negligible impact. Active expiration has two modes:

Fast Cycle

Executed in the beforesleep phase of the event loop (triggered from server.c → main). Its allowed execution time is 1000 + 1000/4 * effort µs, default 1000 µs and capped at 3250 µs when active_expire_effort=10. The fast cycle runs only if the previous slice finished and the percentage of stale keys exceeds config_cycle_acceptable_stale (default 10 %). Under normal settings the fast cycle blocks commands briefly; with active_expire_effort=10 the block can reach 3.25 ms.

Slow Cycle

Scheduled by databasesCron() via the ae framework every 1000/server.hz ms (default 100 ms). Its execution time is governed by config_cycle_slow_time_perc = 25 + 2 * effort % of the event‑loop time, ranging from 25 % (25 ms) to 43 % (43 ms). During the slow cycle Redis scans each database, samples keys, and deletes those that are expired. If a large batch of keys expires together, the slow cycle can block the event loop for up to 25 ms, causing a noticeable performance drop.

Request Penetration to the Database

Massive simultaneous expirations increase request latency; the application falls back to querying the database, raising database load and triggering client retries, which further amplifies Redis CPU usage.

Solution Ideas

To avoid a bulk expiration at midnight, two redesign options were evaluated for keys whose expiration time is tightly coupled to business logic.

Store the real expiration timestamp inside the cache value and add a random extra TTL.

Add a time‑based suffix to the cache key.

Approach 1: Store Real Expiration in Value

Each key receives a random extra TTL (original TTL + random offset). The cached value also contains a field recording the actual expiration timestamp. Application code reads this field to decide whether the entry is still valid.

Advantages: distributes expirations over time, preventing a concentration of deletions.

Disadvantages: requires substantial code changes—modifying the cached value’s data structure and adding expiration‑validation logic.

Approach 2: Add Time Suffix to Key

The key receives a random extra TTL and a date suffix (e.g., user:lottery:2023-10-16). On the first day a user accesses the cache, the key includes that day’s date; the next day a new key with the new date suffix is used, making the previous key logically expired while Redis still removes it based on its TTL.

Advantages: same distribution benefit as Approach 1, but with minimal code changes and no alteration to the cached value’s structure.

Disadvantages: requires precise time synchronization across distributed services and may temporarily double memory usage for keys during the overlap period.

Result

After deploying either redesign, the Redis CPU usage during the midnight peak dropped dramatically, eliminating the 100 % CPU condition and restoring normal latency.