Technical Overview of JD's New Business Intelligence Platform: Offline OLAP, Real‑time Data, and Visualization Solutions

In 2018 JD upgraded its sole data platform, JD Business Intelligence, to unify metrics across sales, traffic, users, products, industry and competitors, and introduced an offline OLAP engine built on ClickHouse, SparkCore and Scala to handle complex multi‑dimensional analyses.

The OLAP solution addresses two main challenges: hourly incremental deduplication of metrics, solved by leveraging ClickHouse's ArrayJoin and sub‑queries to reduce RPC calls, and massive‑scale deduplication, tackled with materialized view roll‑ups and cardinality‑estimated fuzzy deduplication to save memory and time.

To meet merchants' demand for early‑morning data, the team re‑architected offline tasks, shortened task chains, pre‑reserved incremental deduplication data, and trimmed inactive SKU data, cutting 174 tasks and roughly 390 execution hours, delivering results by 7 am instead of noon.

For data product visualization, a generic front‑end component library and skin library were built with React, alongside a timeliness component on the BDP big‑data platform, enabling fast, consistent UI and real‑time/offline analysis across various data products.

The project was recognized as the winning team of the 7th JD Technology Golden Award, which honors outstanding R&D contributions within JD.