From DIKW to Distributed Data Warebase: Evolution of Data Systems and AI‑Driven Architecture

The author starts with the human DIKW (Data‑Information‑Knowledge‑Wisdom) model and maps each layer to its counterpart in the computer world, explaining how raw bits become data, then information via data models, knowledge via embedding vectors, and finally wisdom through advanced reasoning.

Using a homestay platform as a case study, the article shows how the data architecture evolves as business grows: initially a simple relational database (MySQL/PostgreSQL) for MVP, then adding NoSQL (MongoDB) for horizontal scaling, Elasticsearch for real‑time search, and data warehouses (Snowflake, Hive, ClickHouse) for BI analytics, highlighting the pain points of data synchronization, latency, and operational complexity.

The piece then discusses the role of traditional AI for offline insights and real‑time decision making, such as price optimization, requiring feature stores, model training, and online inference.

With the rise of generative AI (e.g., ChatGPT), the article outlines four ways to combine business data with large models: in‑context learning, vector search, retrieval‑augmented generation (RAG), and model fine‑tuning, illustrating each with examples from the homestay scenario.

To address the accumulated challenges, ProtonBase proposes a new generation data system called Distributed Data Warebase, which merges the capabilities of traditional databases and data warehouses, supports relational, JSON, and high‑dimensional vector types, and offers distributed transactions, rich indexing (including inverted and vector indexes), columnar storage, vectorized execution, and materialized views.

This system aims to simplify architecture, reduce data‑sync overhead, and enable AI‑driven intelligence, positioning data systems to let data “emerge intelligence” as the next mission.