Design and Practice of Tencent Lighthouse Fusion Analysis Engine

Tencent Lighthouse is an end‑to‑end data product suite that enables product, R&D, operations, and data science teams to make trustworthy, timely decisions within 30 minutes, driving user growth and retention.

The platform faces a massive, real‑time, and customizable data triangle problem, prompting the need for a next‑generation analysis engine capable of directly accessing detailed data (ODS/DWD) with excellent performance.

The overall technical architecture of the Lighthouse Fusion Analysis Engine consists of a service layer (query, reception, governance), a compute layer (enhanced open‑source engines), a materialized storage layer (Alluxio‑based block cache and ClickHouse acceleration), a storage layer (both managed and unmanaged data sources), an analysis strategy center (workload governance and metric collection), and a productization center (engine can be offered as a standalone product).

Challenge 1 – Fusion : The front‑end, built on Calcite, provides centralized SQL parsing, validation, optimization, and plan generation, while the back‑end fuses multiple engines (Presto, Impala, ClickHouse, etc.) to solve compute‑level integration.

Solutions include a generic MPP engine with high‑performance connectors, an enhanced JDBC connection that splits table scans, push‑down of projection/aggregation/predicate, and engine‑specific optimizations for ClickHouse.

Workload Management (WLM) automatically collects query profiles from different engines, combines them with historical queries, and applies expert‑derived optimization hints (e.g., converting broadcast joins to shuffle joins) to improve resource usage.

Kernel Optimizations address large‑query resource consumption. For Impala, IO scheduling and index utilization (PageIndex, Z‑order, Hilbert) are enhanced. For Presto, HA coordinator, multi‑cluster federation, and disaggregated coordinator with a ResourceManager are explored. For Kudu, source‑level improvements reduce master memory consumption by tenfold.

Acceleration includes multi‑level caching (pre‑computation, unified cache, kernel cache, Alluxio hot‑data cache), a BI engine that routes hot tables to ClickHouse for millisecond‑level query latency, and modern materialized views that automatically rewrite SQL to use the most beneficial view without user intervention.

Practice Summary : The engine introduces numerous innovations across SQL, compute, and storage layers, as illustrated by the key optimization points diagram.

Future Evolution : The roadmap continues to focus on fusion, kernel optimization, and acceleration to achieve direct, high‑performance data access.