Scaling Real-Time Stock Market Data with Redis, Lua, and Go Goroutines

1. Stock Market Data Basics

Stock market quotes are a typical time‑series dataset, similar to log data in IT systems. Real‑time quote services are essential for both retail investors and quantitative traders, requiring ultra‑low latency and high accuracy.

2. Massive Indicator Computation

Daily trading generates billions of indicator calculations (e.g., K‑line candles, financial ratios, multi‑dimensional sorting). For example, with 10 time‑periods and roughly 100,000 securities, the system must handle around 10 × 10⁵ calculations per second, reaching over 1 billion operations per day.

3. In‑process “Compute‑in‑Place” with Redis+Lua

The initial solution stores time‑series data in a Redis cluster and uses Lua scripts executed on each node to compute indicators directly where the data resides. This approach simplifies architecture by keeping storage and computation together, but Redis is single‑threaded and heavy Lua workloads can cause noticeable latency and scaling challenges.

4. Out‑of‑process “Massive Operators” with Goroutine

To overcome Redis‑centric limitations, the computation is moved to dedicated Go services. Each incoming tick snapshot is handed off to a Goroutine that calculates the required indicators and writes results back to Redis. Go’s lightweight Goroutine model allows tens of thousands of concurrent workers with minimal overhead.

5. Comparison of the Two Solutions

Redis‑in‑process : Simple architecture, but each Redis node must compute the same indicator for primary and replica, leading to duplicated CPU work, higher latency (hundreds of ms), and difficulty scaling during market peaks.

Goroutine‑out‑of‑process : Redis acts only as a fast in‑memory store; computation is offloaded to Go workers, reducing Redis CPU load, enabling elastic scaling by spawning more Goroutines, and achieving millisecond‑level latency.

Conclusion

The evolution from an integrated Redis+Lua design to a decoupled Goroutine‑based architecture illustrates how modern coroutine and parallel‑processing techniques can dramatically improve hardware utilization and latency for high‑frequency, large‑scale time‑series services.