Redis Connection Pool Saturation: A Debugging Tale

The story begins with Ke Na saying: "Shuai Fei, the Redis connection pool is full again." I quickly logged into machine 159, restarted it, and replied in the group that Redis was fine.

The problem needed solving; each module's Redis max connections were set to 500, and with roughly 60 × 3 = 180 clusters across three environments, we would need about 80,000 connections, which is clearly impossible.

I edited the deployment script to change the limit from 500 to 50, reducing the required connections to 8,000, and thought the issue was solved.

"Shuai Fei, Redis connection pool is full again," Ke Na shouted again, and I responded that I had already changed it, but the problem persisted.

One day a clue emerged: Brother Gang suggested checking which port was using Redis.

I discovered that machine 79 was occupying almost all of the connection pool.

It turned out a service was hogging the connections, which was quite annoying.

I remembered that netstat could reveal this, so I gave it a try.

At that time the issue could no longer be reproduced; I saw the 812 process calling 192.168.186.159, and after counting the connections I found a huge usage.

The count was 16,000, meaning almost all connections were taken by that process.

Running ps aux| grep 812 revealed the user zz es.

I contacted the responsible RD, who admitted the service was consuming a large number of connections and not releasing them.

After stopping the service, Redis connections dropped to 500, a reduction of about 16,000.

The case was fully resolved.

Original article by QA author Wang ShuaiFei, 2017-01-11