Log Analysis and Schema‑On‑Read: Design and Implementation of the Honghu Real‑Time Heterogeneous Data Platform

The article begins by outlining the importance of log analysis for both consumer (e.g., e‑commerce recommendation) and enterprise scenarios (e.g., process mining, IT operations), and highlights technical challenges such as high ingestion rates, diverse and evolving log formats, and the need for both batch and ad‑hoc queries.

It then contrasts two technical approaches: Schema‑On‑Write (traditional ETL, requiring predefined tables and schemas) and Schema‑On‑Read (also called schema‑on‑read or ELT, where raw logs are stored and schemas are derived at query time). The advantages of schema‑on‑read—flexibility, lower storage cost, and faster ingestion—are explained, along with its trade‑offs compared to schema‑on‑write.

The piece introduces the Honghu platform (by Yanhuang Data) as a free, one‑stop solution for heterogeneous real‑time data analysis. It describes Honghu’s storage model (schema‑less event sets, time‑stamp indexing, inverted indexing, column‑ariented shards, and automatic handling of schema evolution) and its compute engine, which offers a schemaless SQL interface, scalar and table functions, CTE support, and materialized view acceleration.

Key technical details include automatic JSON/key‑value parsing, field extraction, dynamic schema union, vectorized execution, JIT‑compiled functions, and the ability to reuse modeling logic via views. The platform also supports pipelines that combine read‑time and write‑time modeling, enabling seamless transition from traditional ETL to schema‑on‑read workflows.

Finally, a Q&A section addresses migration strategies from ETL to schema‑on‑read, automatic type inference, and use cases such as IoT sensor data, emphasizing that both approaches can complement each other in modern big‑data environments.