From Single‑Node to Distributed: The Evolution of Modern Database Services

In the 1950s Grosch’s law and Moore’s law highlighted the exponential growth of computing power, yet many modern CPUs remain under‑utilized, prompting the rise of distributed computing to harness idle resources.

Evolution of Database Technologies

The lecture reviews the origins of Redis and MongoDB, noting their rapid ascent in the DB‑Engines rankings—Redis dominates the key‑value space while MongoDB leads the document‑oriented category. Both emerged around 2009, improving on earlier systems like Memcached by introducing richer data structures (lists, hashes, sorted sets) and a JSON‑centric storage model.

Design Choices Behind Distributed Databases

Three primary architectural approaches are discussed:

Client‑side routing (e.g., TDDL): The client parses SQL and directs queries to the appropriate backend, offering simplicity but potentially high connection counts.

Proxy layer: A middle‑tier intercepts traffic, handling routing and protocol translation, which eases management and language‑agnostic SDK reuse, yet struggles with complex transactions.

Compute‑storage separation: Storage is abstracted from compute nodes, enabling independent scaling, faster replica provisioning, and higher IOPS, as exemplified by Amazon Aurora, F1/Spanner, and TiDB.

The speaker explains how compute‑storage separation reduces replication latency, simplifies backup via block‑level snapshots, and improves high‑availability failover, while also noting limits such as finite storage capacity and increased engineering complexity.

Comparative Advantages and Drawbacks

Client‑side solutions excel in performance and multi‑source support but increase connection overhead and SDK maintenance. Proxy solutions are easier to upgrade and maintain language compatibility but lack robust distributed transaction support. Compute‑storage‑separated systems deliver superior IOPS and cost‑effective scaling, though they face challenges with SQL compatibility and storage limits.

Q&A Highlights

Key questions addressed include why cloud storage has capacity limits (metadata management and network bandwidth constraints), the existence of JD Cloud’s MySQL‑compatible distributed storage engine (DRDS), and the relationship between distributed caches and databases, emphasizing that caches are simpler due to the absence of transaction and complex query requirements.

Overall, the session provides a comprehensive roadmap of database service evolution, from single‑node designs to sophisticated distributed architectures, and offers practical insights for selecting the appropriate solution based on workload characteristics.