How to Build Real‑Time Data Pipelines for E‑Commerce Promotions

Real‑Time Data Collection

To satisfy e‑commerce promotion monitoring, the following data sources must be ingested continuously:

Buyer search logs

Product view records

Order details

All website traffic (PV/UV)

Machine metrics (CPU, MEM, I/O)

Application logs

Real‑Time Computation

After collection, the stream is processed to produce business‑critical metrics:

Total sales amount (GMV) across all products

Top‑5 selling products

Join of user behavior (search, view) with order events

Count of requests per IP address

Per‑minute averages and 75th‑percentile values for CPU/MEM/I/O

Filter and forward only ERROR level log entries

Real‑Time Delivery

Computed results are dispatched to downstream consumers via two main paths:

Alert channels : email, SMS, DingTalk, WeChat. The computation layer compares metrics against configurable thresholds and triggers alerts when thresholds are exceeded.

Storage back‑ends : message queues, relational/NoSQL databases, file systems. Dashboards (e.g., Elasticsearch, HBase) query these stores to display up‑to‑date metrics for operations, monitoring, development, and management.

Typical Real‑Time Scenarios

Traffic signal data

Road congestion statistics

Public‑security video monitoring

Server health monitoring

Financial market risk calculations

Real‑time ETL

Fraud detection for banks/payments

Additional Flink user surveys report use cases such as real‑time analytics, metric aggregation, reporting, CEP‑based decision making, ad‑tech multi‑stream joins, industrial IoT, and log processing.

Four Core Real‑Time Use‑Case Categories

Real‑time data storage with micro‑aggregation, field filtering, and data masking.

Real‑time data analysis, often feeding machine‑learning models for recommendation.

Real‑time monitoring and alerting for finance, traffic, servers, and logs.

Real‑time reporting, e.g., sales dashboards and Top‑N product displays.

Batch vs. Stream Processing

Batch (offline) processing handles large, fixed datasets over long windows (daily, weekly, monthly). Jobs are scheduled, may involve complex transformations, and the input data does not change during execution.

Stream (real‑time) processing ingests continuous, unordered, high‑volume data. It must emit results with low latency, often using sliding or tumbling windows, and cannot assume a bounded input.

Characteristics of Real‑Time Streams

Data arrives instantly and may be out‑of‑order.

Volume is large and unpredictable.

Processed data is typically not re‑readable without costly recomputation.

Advantages of Real‑Time Computing

Enables immediate alerts, short‑window aggregations, multi‑dimensional correlation, and dynamic personalization (e.g., “千人千面” recommendations). Stakeholders receive up‑to‑date insights for operational decision‑making.

Challenges of Real‑Time Stream Processing

Guaranteeing processing semantics: exactly‑once, at‑least‑once, or at‑most‑once.

Maintaining timely processing under bursty ingest rates to avoid backlog.

Scaling processing and storage layers dynamically with workload fluctuations.

Providing fault‑tolerance and high availability for both compute and storage components.