Event Bus: Architecture, Technical Challenges, and Solutions for High‑Throughput Data Standardization

The event bus, also called a data pipeline, is a core component of the risk insight platform that standardizes real‑time risk data, normalizes code logic, and supports high‑throughput parsing, plug‑in I/O, and low‑code hot‑loading, thereby enabling efficient data processing and risk management.

1. Event Bus Introduction – It unifies diverse upstream data sources (e.g., JMQ2, JMQ4, FMQ) and transforms complex messages into a flat key‑value structure through a three‑layer architecture: source , transform , and sink . The system supports Groovy, Python scripts, and custom JAR uploads, allowing one‑way ingestion and multi‑direction output.

2. Technical Challenges – The platform faced data structure diversity, repetitive serialization code, inefficient parsing/writing, varied input/output components, and rapidly changing business requirements, leading to high maintenance costs and throughput bottlenecks.

3. Solution Overview

Unified data standardization: all downstream data is emitted as a flat key‑value map.

Code‑logic normalization: use JsonPath to generate a context object for field extraction.

High‑throughput parsing/writing: single parse with multi‑route sinking, configurable batch size and frequency per warehouse.

Plug‑in I/O: easily add new source or sink components.

Low‑code hot‑loading: script‑based configuration with hot reload and function plug‑ins.

4. Architecture – The event bus consists of three layers:

Source : data ingestion (JMQ2, JMQ4, FMQ, etc.).

Transform : core processing, including parsing, filtering, enrichment, and conversion operators; ultimately produces a standardized key‑value record.

Sink : data output to message queues or storage systems such as ClickHouse, Elasticsearch, R2M.

5. Core Class Diagram – The top‑level interface IEventHubExecutor defines execute . Sub‑interfaces IEventHubParse , IEventHubTransform , and IEventHubSink correspond to source, transform, and sink, enabling plug‑in adapters for different middleware (e.g., extending to a Jes sink for Elasticsearch).

6. Custom Functions – Implemented via the Aviator expression engine. Functions can be defined in Groovy/Java scripts or uploaded as JARs, compiled into AvaitorFunction instances, registered in the Aviator context, and invoked through a FunctionExecutor .

7. Dynamic Grouping, One‑Click Degradation, and Traffic Monitoring

Dynamic grouping separates physical (machine‑based) and logical (consumption‑group‑based) partitions, allowing flexible traffic allocation.

One‑click degradation supports discard‑type (data dropped) and backlog‑type (data queued) strategies for high‑traffic periods.

Traffic monitoring is integrated with UMP; each critical point registers profiling info, e.g., Profiler.registerInfo(this.getClass().getSimpleName(), UmpUtil.UMP_APP_NAME, false, true); .

8. Future Outlook – The event bus has handled up to 5,000 w/min during peak promotions and 2,000 w/min daily. Future work includes exploring Flink‑CDC integration and further performance optimizations.

Conclusion – The event bus provides a robust, extensible, and high‑throughput solution for real‑time risk data processing, offering standardized pipelines, plug‑in flexibility, and operational observability that are essential for modern risk‑insight platforms.