How to Build a High‑Availability Real‑Time Logistics Dashboard with Flink and ClickHouse

Overview

In logistics systems, the supply‑chain dashboard is complex, with long data chains and high stability requirements; after two years of iteration it has become mature and stable.

Key Technologies

1. Flink data processing pipeline and OLAP engine

We use real‑time Flink combined with ClickHouse to build the base model, enabling one‑click switching via dual streams and EasyData.

2. Metric consistency

Processing and presentation are separated; atomic warehouse metrics can be rolled up to regional and national levels, ensuring dimensional and temporal consistency, with caching layers for various scenarios.

3. Stability construction

Data link stability

Service fallback

Metric validation mechanisms

Recalculation mechanisms

Various issues have been encountered, such as CK partition blocking, network failures, Flink message loss, checkpoint failures, and complex logic causing back‑pressure.

Background

The supply‑chain dashboard supports real‑time monitoring during major promotions, providing over 170 metrics refreshed every minute from multiple data sources.

Solution

2.1 Data model storage selection

We compared Elasticsearch and ClickHouse; ClickHouse offers superior write and query performance for OLAP workloads, though Elasticsearch may be used when high concurrency or full‑text search is needed.

2.2 Overall architecture

The architecture consists of five layers: model processing, data processing, single‑warehouse metric processing, regional metric processing, and presentation, each with defined responsibilities.

2.3 Metric layering and consistency design

All warehouse‑order metrics share a single processing logic; after the primary task finishes, higher‑level metrics are derived, ensuring data consistency. Caching tables improve query performance for different reports.

2.4 Stability design

We isolate the Flink‑ClickHouse link with dual streams for quick failover. Common failure points include CK partition overload, rename delays, network outages, message loss, checkpoint failures, uneven sharding, missing dimension tables, and complex operator back‑pressure.

2.5 Extensibility design

Configuration‑driven UCC adapts to business needs, covering 4H/24H/28H modes, threshold switches, whitelist for auto‑refresh, historical date selection, and recalculation mechanisms.

{
  "thresholdEnable": "false",
  "upperLimit": "1.6d",
  "lowerLimit": "0.6d",
  "zyShowFlag": true,
  "swShowFlag": true,
  "jjShowFlag": true,
  "wdShowFlag": true,
  "todayTradeCleanRateShowFlag": true,
  "promotionTradeCleanRateShowFlag": true,
  "isDebug": true,
  "isCacheOn": true,
  "isWriteMinuteAndHour": true,
  "isMinuteWrite": true,
  "isHourWrite": true,
  "isMinuteNotice": false,
  "isHourNotice": false
}

Comparison strategy configuration

{
  "sTime": "2023-06-17 00:00:00",
  "eTime": "2023-06-17 19:59:59",
  "tbSTime": "2022-06-17 00:00:00",
  "tbETime": "2022-06-17 19:59:59",
  "hbSTime": "2022-11-10 00:00:00",
  "hbETime": "2022-11-10 19:59:59",
  "showType": "24h",
  "special24hCompDateStr": "2022-06-17",
  "specialCompDateStr": ""
}

2.6 Data monitoring

We combine automated front‑end snapshots, packet capture, minute‑level database logging with Grafana dashboards, and JD Power testing models to verify and troubleshoot data accuracy.