Modern Data Paradigms: From Relational Databases to Vector Retrieval and AI

The article begins by introducing the author, Shen Lian, a database kernel leader at Ant Group, and sets the stage for discussing how modern data technologies, empowered by big data and large models, release massive technical dividends and free applications from data shackles.

It then explains the 4V properties of modern data—Volume, Velocity, Variety, and low Value density—and how these characteristics challenge traditional data architectures, requiring upgrades to storage, retrieval, and analysis capabilities.

Relational databases are described as the cornerstone of the information age, highlighting their ACID guarantees, SQL query language, and the historical development from Codd's relational model to products like Oracle, MySQL, and SQL Server.

The rise of NoSQL is examined, noting the limitations of relational systems for massive, heterogeneous workloads and introducing KV, wide‑column, document, graph, and time‑series models. Key NoSQL systems such as Dynamo, BigTable, HBase, MongoDB, Neo4j, and InfluxDB are discussed, along with their design goals of horizontal scalability and flexible data modeling.

Polyglot Persistence and CQRS are presented as architectural patterns that combine relational and NoSQL stores, allowing write operations to use strong consistency while read paths leverage scalable, schema‑less systems.

The article moves to big‑data analytics, describing Lambda architecture (offline batch + online real‑time layers) and its application in Ant’s risk‑control scenario.

It then shifts focus to unstructured data, explaining that embeddings—vector representations generated by AI/ML models—provide a unified semantic representation for text, images, audio, and video, enabling similarity search.

Vector retrieval techniques are detailed, contrasting exact k‑Nearest Neighbor (kNN) with Approximate Nearest Neighbor (ANN) methods such as IVF, HNSW, PQ, and IVFPQ, and discussing distance metrics (Euclidean, cosine, inner product) and index structures.

Three product forms for vector search are listed: dedicated vector databases, vector‑enabled regular databases, and vector search libraries.

Hybrid retrieval is introduced as the combination of traditional scalar/full‑text search with vector search, requiring re‑ranking algorithms (RRF, RankNet, LambdaRank, LambdaMART) to merge results.

Retrieval‑Augmented Generation (RAG) is described as the integration of search with large language models, allowing up‑to‑date, domain‑specific knowledge to be injected into LLM outputs, with frameworks like LangChain and LlamaIndex providing ready‑made stacks.

Finally, the article summarizes three data paradigms across eras: relational databases for the information age, NoSQL for the big‑data era, and LLM + VectorDB for the generative AI era, each removing a different set of data constraints and driving the evolution of modern applications.