Full-Chain Load Testing: Redis Large-Key and Memory Leak Issues and Solutions

During large-scale promotional events, e‑commerce systems must remain stable. Full‑chain load testing using a shadow database replicates production traffic without contaminating real data.

Redis Large‑Key Problem

During load testing of the order confirmation page, QPS could not increase and response time (RT) rose above 1 s. Tracing revealed a Redis call with high latency. The Redis node had high bandwidth usage because a single large key (megabytes in size) was being read concurrently, saturating the node’s bandwidth.

Solution:

Split the large key into multiple smaller keys (key sharding) to reduce payload size per request.

Introduce a local cache to avoid frequent Redis reads, using a message‑based invalidation strategy to keep data consistent.

Memory Leak Problem

After several load‑test rounds, the service’s memory usage kept rising and did not drop after garbage collection. Analysis showed that ApplicationShutdownHooks were added on every request instead of once, causing an ever‑growing number of hook objects.

Solution:

Ensure the shutdown hook is registered only once. Example before and after:

Before:

public static ExecutorService getCartsListExecutor() { shutdownHandle(cartsListForkJoin, "cartsListForkJoin"); return cartsListForkJoin; }

After:

static { shutdownHandle(cartsListForkJoin, "cartsListForkJoin"); shutdownHandle(cartsCacheInitForkJoin, "cartsCacheInitForkJoin"); }

High Machine Memory Usage

Even after fixing the leak, JVM committed memory stayed around 9 GB (Xmx = 9600 MB), causing the host memory to stay above 70 % and trigger alerts. The committed memory does not shrink after GC.

Solutions:

Increase physical memory (e.g., from 16 GB to 32 GB).

Reduce the maximum heap size (e.g., lower Xmx) to limit committed memory.

Adjust monitoring thresholds (e.g., raise alert level to 80 %).