How a Single Redis Command Cost a Company $4 Million – Risks and Prevention

1. Redis Dangerous Command Caused $4 Million Loss

Recently, a PHP engineer executed a risky Redis command (e.g., keys wxdb…cf8 ) in production, locking Redis, spiking CPU, and causing payment pipelines to stall; after seconds, traffic shifted to RDS, triggering a database avalanche and a $4 million loss. The company warned that repeat offenses would lead to termination and a gradual revocation of operational permissions.

2. Performance Test with 10 Million Records

1. Write Script File

Generate 10 million key‑value pairs:

for((i=1;i<=10000000;i++)); do echo "set k$i 哪吒编程$i" >> /tmp/test1.txt; done;

Verify the file /tmp/test1.txt contains the data.

2. Import 10 Million Records into Redis

cat /tmp/test1.txt | redis-cli -a 111111 --pipe

3. Verify with keys *

4. Disable keys * via Configuration

Add the following lines to redis.conf:

rename-command keys ""
rename-command flushdb ""
rename-command flushall ""

3. Replace KEYS * with SCAN

The SCAN command iterates keys using a cursor, returning a new cursor and a batch of keys each call; iteration ends when the cursor returns 0.

Syntax:

SCAN cursor [MATCH pattern] [COUNT count]

4. Reject Big Keys

1. Alibaba Cloud Redis Development Guidelines

Big keys are prohibited to avoid network congestion and slow queries. Keep string values under 10 KB and limit hash/list/set/zset elements to 5 000.

2. Deleting Big Keys

Use DEL for string keys.

For other types, employ incremental deletion with HSCAN, SSCAN, or ZSCAN, and avoid automatic expiration that blocks the main thread.

Example for hash deletion:

public void delBigHash(String host, int port, String password, String bigHashKey) {
    Jedis jedis = new Jedis(host, port);
    if (password != null && !"".equals(password)) {
        jedis.auth(password);
    }
    ScanParams scanParams = new ScanParams().count(100);
    String cursor = "0";
    do {
        ScanResult<Entry<String, String>> scanResult = jedis.hscan(bigHashKey, cursor, scanParams);
        List<Entry<String, String>> entryList = scanResult.getResult();
        if (entryList != null && !entryList.isEmpty()) {
            for (Entry<String, String> entry : entryList) {
                jedis.hdel(bigHashKey, entry.getKey());
            }
        }
        cursor = scanResult.getStringCursor();
    } while (!"0".equals(cursor));
    // Delete the big key itself
    jedis.del(bigHashKey);
}

3. Problems Caused by Big Keys

Uneven memory distribution and difficult cluster migration.

Timeout deletions that block threads.

Network traffic congestion.

4. Detecting Big Keys

Use redis-cli --bigkeys or compute per‑key memory usage with memory usage key: