MaxCompute Semi-Structured Data Solutions: Architecture, Comparison, and Performance Benefits

First, the article defines semi‑structured data as a middle ground between structured and unstructured data, highlighting its self‑describing nature and typical formats such as JSON and XML, which combine flexibility with a protocol for parsing.

It then contrasts structured data—strictly defined tables with fixed schemas—and unstructured data—free‑form media lacking any inherent schema—explaining the trade‑offs in flexibility, storage efficiency, and query performance.

Semi‑structured data inherits the flexibility of unstructured data while providing enough schema information to enable efficient parsing and access, making it suitable for logs, IoT telemetry, mobile event reporting, and autonomous driving.

The article reviews traditional data‑warehouse solutions: schema‑on‑read , which stores raw data and parses it at query time (high flexibility, low performance), and schema‑on‑write , which parses data during ingestion (lower flexibility, higher performance). It discusses the maintenance costs of schema‑on‑write in fast‑changing business environments.

MaxCompute’s semi‑structured data solution is then presented. MaxCompute is a serverless, cloud‑native data‑warehouse service that supports massive scale (hundreds of GB to EB). It ingests semi‑structured data, stores it using the AliORC columnar format, and leverages dynamic schema extraction to columnar‑store stable fields while preserving flexibility for evolving fields.

The solution extracts common structures from short‑term stable data, column‑stores them to reduce storage and improve query speed, and handles long‑term schema evolution with adaptive parsing. It also addresses dirty data by storing values together with their type information in a binary format, and mitigates sparse data by aggregating low‑frequency fields into a special column.

During query execution, the engine performs column pruning, detects the actual runtime type of each field, and inserts dynamic conversion operators as needed to unify types (e.g., converting strings or binaries to integers) before applying filters, thus achieving adaptive processing.

Performance analysis shows that the columnar JSON approach yields near‑native columnar query speeds and significantly reduces storage compared with raw JSON strings, while still leaving room for further optimization (e.g., better handling of date types).

In conclusion, MaxCompute’s out‑of‑the‑box semi‑structured columnar solution provides high performance, low storage cost, and minimal operational overhead, while supporting dynamic schema evolution, dirty and sparse data handling, and adaptive query processing.