Stability Challenges and Engineering Solutions for an Inventory Platform

Stability Challenges of the Inventory Platform

The inventory platform provides end‑to‑end stock management across the order lifecycle, but during construction it encountered several stability issues: numerous inter‑dependent business processes, complex workflows that are error‑prone, strict inventory data accuracy requirements, database hotspot contention during flash‑sale or live‑stream events, and high‑frequency, large‑scale calculations for shop inventory redistribution.

Stability Construction Measures

Traffic Splitting

Traffic was categorized into core flows that must be highly available, large‑scale batch operations, and non‑real‑time data sync. Different service groups were created to handle each category, allowing tailored timeout configurations and isolation of heavy‑weight operations.

Gray‑Release Links

Instead of embedding numerous feature switches, a merchant‑based gray‑release link was introduced, enabling gradual rollout and rollback of changes without adding extra control code, thereby reducing maintenance overhead and online incidents.

Operation Quantity Verification

For multi‑record inventory operations, a verification step ensures that each record receives the correct operation quantity and that change logs are generated accordingly.

Database Hotspot Mitigation

Redis caching was employed to offload hotspot inventory deductions, boosting pre‑allocation TPS from 50 to 1,200 (24× increase) and reducing TP99 latency from 3,000 ms to 130 ms. Consistency between Redis and the database is maintained via DB‑level locking, Redis transactions, and MQ‑based retry mechanisms.

Consistency Checks and Monitoring

Daily millions of inventory operations trigger automated consistency checks between DB and Redis, with discrepancy logs stored in Elasticsearch. Management pages allow querying and correcting data by merchant, product, or order.

Async Rate Limiting for Hotspots

A sliding‑window algorithm detects hotspot inventory and applies asynchronous rate limiting, implemented via AOP interceptors, to smooth traffic and prevent CPU overload.

Shop Inventory Stability Enhancements

Pre‑emptive identification of 25 trigger points for inventory changes led to targeted CPU usage governance and JSF service isolation, reducing resource contention and improving service availability.

Future Plans

Plans include richer business‑level monitoring alerts, hourly data comparison for anomaly detection, and development of an automated DB‑Redis inconsistency comparison tool to accelerate root‑cause analysis.