How Real-Time Big Data Streaming Powers Double 11 E‑Commerce Success

Background and Motivation

Every year the Double 11 shopping festival pushes e‑commerce platforms to their technical limits, demanding unprecedented sales, transaction, and payment peaks. Behind these numbers lies massive real‑time data processing, which NetEase Kaola leverages to turn raw event streams into actionable business insights.

Continuous Data Generation and Real‑Time Computing

In the real world, data is produced continuously—user clicks, GPS updates, sensor logs, media uploads, etc. This creates a data stream that can be captured and processed instantly. Real‑time computation updates results as new events arrive, enabling applications such as live weather forecasts or dynamic recommendation engines.

Practical Applications at NetEase Kaola

Custom 1: A large screen visualizes live sales totals, category ratios, order trends, and user geography during Double 11, updating instantly with each transaction to guide marketing decisions.

Custom 2: Financial risk control uses streaming analytics to match user behavior against risk models, detect anomalies, assign risk levels, and trigger automated alerts and workflow changes.

Custom 3: Real‑time recommendation systems continuously refine user interest profiles, delivering personalized news, music, or product suggestions based on the latest interactions.

From “Store‑Then‑Compute” to “Compute‑While‑Storing”

Traditional batch processing follows the pattern store → query → output, which is too slow for massive, continuously arriving data. Incremental streaming computes results on new records, merges them with historical aggregates, and stores only the updated results, dramatically reducing latency and storage pressure.

Key Technical Challenges

Distributed real‑time streaming must achieve low latency, high throughput, and fault tolerance. Data ingestion pipelines need buffering and high‑speed transport; computation nodes must perform incremental aggregation; output modules must compress and batch results. Accurate incremental results require careful handling of data retractions and state updates. System failures must be detected and recovered quickly without breaking the incremental model.

NetEase’s Sloth Stream Computing Platform

Sloth addresses these challenges by providing a multi‑tenant, easy‑to‑use platform that abstracts away low‑level distribution details. It supports SQL‑based stream processing, allowing developers to write familiar queries that are automatically optimized for incremental execution. The platform integrates with various storage and messaging systems, enabling seamless data flow from source to sink.

SQL for Stream Processing

Just as SQL transformed batch analytics, stream‑SQL treats a continuously growing data stream as a dynamic table. Queries run indefinitely, emitting results as the underlying data evolves. This model simplifies development, reduces the need for custom code, and aligns with existing data‑warehousing skills.

Future Directions

Beyond text, streaming will extend to images, audio, IoT telemetry, and online machine learning. Automatic optimization of stream‑SQL engines, support for complex windowing, and handling of output triggers are active research areas that will further broaden real‑time analytics.