Real-Time Risk Insight: Architecture Evolution and Future Outlook

Introduction – The session, presented by JD Technology architect Meng Xiangtao, focuses on the evolution and thinking behind real‑time risk insight architecture.

Challenges of Real‑Time Risk Insight – Three main problems are identified: data silos across multiple systems, massive data volume requiring real‑time processing, and the need for real‑time value extraction to support business decisions.

Platform Vision – JD aims to provide a monitoring and intelligent analysis product that enables fast data onboarding, comprehensive risk monitoring, automatic anomaly detection, and rapid root‑cause attribution.

Architecture Evolution

1. Version 1.0 – Simple, Flexible but Limited Extensibility – A four‑layer design (data ingestion → ES cluster → metrics, scheduling, dashboards → risk analysis/monitoring) suffered from poor universality, hard‑coded data processing, and performance bottlenecks of ES.

2. Version 2.0 – Platform‑centric, Component‑based, Plug‑in – Introduced a four‑layer stack: data warehouse, compute engine (Flink), data modeling, and analysis/pre‑warning. Implemented an event‑bus with SPI‑based connectors, modular storage (ClickHouse, ES, HBase, MQ), and low‑code SQL generation.

3. Version 3.0 – Real‑Time Data Warehouse + Intelligent Algorithms – Added a three‑layer data model (ODS → DWM → DWS) for lightweight aggregation, separated ingestion, computation, and storage pipelines, and integrated algorithm services (anomaly detection, attribution) via an algorithm gateway.

Core Components

• Event Bus – Standardizes risk data handling (source → transform → sink) with extensible connectors, operators, and function plugins (Groovy, Avitor, JSONPath).

• Plug‑in Architecture – Five‑layer data link (event bus, storage cluster, SQL engine, data source engine, intelligent analysis) enabling interchangeable components such as ClickHouse, Redis SQL, and custom drivers.

• Anomaly Detection Service – Provides lightweight models that learn normal metric trends, generate anomaly scores, and support multi‑dimensional event analysis for risk perception.

Future Thinking and Outlook – Plans include scenario‑specific analysis (e.g., marketing, order), expanding from single‑metric to multi‑metric alerts, shifting from individual to group analysis, and adopting lake‑warehouse integration to unify batch and streaming data.

Q&A Highlights

• Reducing reliance on expert experience by using anomaly models to auto‑learn thresholds.

• Extending the architecture to support real‑time decision‑making (e.g., blocking risky transactions) via a closed‑loop pre‑warning and interception list.

• Managing false‑positive rates by adjusting anomaly scores and keeping model computation lightweight.