How We Boosted Inventory Platform Stability 24× with Smart Traffic Splitting and Redis Caching

Stability Challenges Faced by the Inventory Platform

The inventory platform provides comprehensive stock management across the entire order lifecycle in e‑commerce, but during construction we encountered several stability issues.

Key Problems

Multiple business processes share the same application, causing interference (e.g., sales order stock pre‑allocation failures also affect return pre‑allocation).

Complex workflows are error‑prone; new staff need a week to understand them.

High accuracy requirements for stock data; a single error can block downstream orders.

Database hotspot operations during flash sales or live promotions lead to lock contention and performance degradation.

High‑frequency, large‑scale calculations for shop inventory can saturate CPU resources.

Stability Measures

Traffic Splitting

We identified three traffic characteristics and split traffic accordingly:

Core traffic (e.g., sales order stock pre‑allocation) that must be highly reliable.

Large‑volume, non‑real‑time traffic such as post‑outbound stock updates.

Batch operations with widely varying batch sizes, requiring different timeout settings.

Gray Release Path

Instead of adding numerous feature switches, we built a merchant‑based gray release path, allowing gradual traffic rollout per merchant, reducing maintenance overhead and online errors.

Operation Quantity Validation

When a product involves multiple inventory records, we validate each record’s operation quantity and generate change logs accordingly.

Database Hotspot Mitigation

We introduced Redis caching for hotspot items, gradually shifting traffic per merchant.

After optimization, pre‑allocation TPS increased from 50 to 1,200 (24×), and TP99 dropped from 3,000 ms to 130 ms.

Data Consistency Between Database and Redis

We added a lock‑db‑plus‑Redis‑transaction initialization flow and an MQ‑retry sync process to ensure consistency.

We also record cache operation logs in Elasticsearch to aid troubleshooting.

Handling Key Merchant Custom Logic

For merchants with heavy custom logic, we applied asynchronous rate limiting and hotspot detection using a sliding‑window algorithm, implemented via AOP.

Shop Inventory Stability

Shop inventory is shared across multiple online and offline channels; real‑time synchronization prevents overselling.

CPU Usage Governance

We pre‑emptively throttled high‑impact calculations, reducing CPU spikes.

JSF Service Governance

We isolated calculation tasks from JSF services, preventing resource contention.

Virtual Bundle Governance

Virtual bundles generate massive calculation workloads; we split MQ and apply JMQ4 rate limiting to smooth processing.

Future Plans

Enhanced Business Monitoring and Alerts

We will add data‑level monitoring to verify that successful API responses also correspond to successful business processing and persistence.

Database‑Redis Inconsistency Comparison Tool

A tool will automate the analysis of mismatched records, reducing manual effort.