How Blink Powers Alibaba’s Real‑Time Supply‑Chain Data Warehouse

Alibaba's Cainiao (菜鸟) began exploring Blink, an internal version of Apache Flink, in April 2017 and adopted it as the primary real‑time computation engine by July 2017 to support its extensive supply‑chain data warehouse.

Retraction Mechanism

Order fulfillment requires real‑time estimation of each logistics node. When upstream data is delayed, downstream estimates must be recomputed, causing a "key change" problem that traditional engines struggle with. Blink’s built‑in Retraction mechanism uses state to emit delete messages for outdated records and then updates them with new values.

Example SQL:

SELECT plan_tms_sign_time,
       SUM(1) AS plan_tms_sign_lgtord_cnt
FROM (
    SELECT lg_order_code,
           LAST_VALUE(plan_tms_sign_time) AS plan_tms_sign_time
    FROM dwd_csn_whc_lgt_fl_ord_ri
    GROUP BY lg_order_code
) ss
GROUP BY plan_tms_sign_time;

Dimension‑Table Join Optimizations

Supply‑chain entities are numerous, leading to heavy dimension‑table joins. Blink improves performance through:

Async IO : asynchronous requests avoid blocking, boosting throughput.

Cache : LRU or ALL cache reduces repeated key lookups.

Invalid‑Key Cache Handling : configurable retention time for missing keys.

Distribute By : partitioning by key improves cache hit rates.

Additional optimizations such as SideInput and Partitioned ALL Cache are under development.

Data Skew Mitigation

Blink addresses data skew in seller, industry, and product statistics using MiniBatch/MicroBatch, LocalGlobal aggregation, and PartialFinal parameters, which automatically disperse hot keys without manual hash‑and‑aggregate code.

Relevant configuration:

# MiniBatch/MicroBatch batch interval (ms)
blink.miniBatch.allowLatencyMs=5000
blink.microBatch.allowLatencyMs=5000
# Maximum records per batch to avoid OOM
blink.miniBatch.size=20000
# Enable LocalGlobal aggregation
blink.localAgg.enabled=true
# Enable PartialFinal for distinct counts
blink.partialAgg.enabled=true

Timeout Statistics

For scenarios where no upstream message arrives (e.g., items not shelved within a time window), Blink can use delayed message delivery, Kafka timeout messages, CEP, or a custom ProcessFunction with timers. The chosen solution combines ProcessFunction timers with a 1‑minute configuration to emit timeout alerts efficiently.

Zero‑Point Optimizations

During major promotions, Blink’s FlushInterval, MiniBatch size/latency, checkpoint strategy, and HBase as the downstream store are tuned to ensure metrics appear on dashboards within seconds after midnight.

Future Outlook

Blink continues to evolve, adding batch support, intelligent resource auto‑tuning, multi‑language APIs (including Python), and broader AI algorithm integration, positioning it as a unified platform for both streaming and batch big‑data workloads.