Observations on the Third Evolution of Data Infrastructure and the Next‑Generation Data Platform Architecture

The rapid evolution of data infrastructure has moved from the database era to the big‑data era and now to the generative AI era, prompting a third technological revolution in data platforms.

Current Data Platform Status – Modern platforms ingest data from production systems, store it in heterogeneous storage (data lakes and MPP warehouses), and expose it to BI and AI workloads. While structured data processing is mature, challenges remain around data redundancy, high operational costs, and handling large volumes of unstructured data.

Next‑Generation Architecture Evolution – Three key directions are discussed: (1) the emergence of a unified lake‑warehouse (lakehouse) that provides a single copy of data for batch, streaming, and interactive analytics; (2) cloud‑native designs that separate storage and compute, leveraging Kubernetes for resource pooling and elasticity; (3) consolidation of multiple compute engines into a single, native engine supporting batch, streaming, and AI workloads.

The proposed reference architecture emphasizes a single compute engine, a unified storage layer based on open table formats, and tight integration with AI models, allowing AI functions to be invoked as first‑class compute primitives.

Future Trends and Open Problems – Four trends are identified: (1) shifting from 1:1 to M:N data‑platform relationships; (2) data‑centric AI where data quality becomes the primary differentiator; (3) a resurgence of search‑oriented architectures for vector‑based retrieval; (4) a ten‑fold increase in the importance and difficulty of unified metadata management for both structured and unstructured assets.

Unresolved challenges include the choice between SQL and Python for autogenerated pipelines, the timeline for fully autonomous data‑platform “auto‑driving,” and the ultimate representation of knowledge extracted from semi‑structured data.

Summary – The talk revisits the evolution of data platforms, shares practical insights from Yunqi Technology on next‑generation lakehouse designs, and outlines future AI‑driven directions and open research questions.