CB‑SQL: JD Retail’s Elastic Distributed Database Built on CockroachDB – Architecture and Implementation Details

Overview

CockroachDB is an open‑source NewSQL database inspired by Google Spanner, capable of managing up to 4 EB of data, providing ACID guarantees, SQL compatibility, multi‑region deployment, automatic horizontal scaling, fault‑tolerance, and both OLTP and lightweight OLAP capabilities.

Architecture

The system follows a layered design: a SQL layer that translates relational queries into KV operations, and a KV layer built on RocksDB (wrapped via CGO in Go). The SQL layer retains traditional concepts such as databases, tables, schemas, rows, columns, and indexes, while the KV layer stores data as key‑value pairs.

Storage

Data is stored in RocksDB instances called Store ; each store manages multiple range replicas (default three). Ranges are split when they exceed ~64 MB and merged when too small. Raft consensus ensures strong consistency across replicas, with a leader handling writes and lease mechanisms preventing split‑brain scenarios.

Cluster Management

Nodes exchange metadata via a Gossip protocol, sharing load metrics, range statistics, and network topology information. This enables autonomous load‑balancing, replica migration, and fault recovery without manual intervention, and the system exports monitoring data (e.g., to Prometheus) via an internal time‑series store and Admin UI.

MVCC

Each key supports multiple versions using a RealKey that concatenates the transaction’s commit timestamp. Versions have a default TTL of 25 hours and are periodically compacted; the TTL can be adjusted for long‑running OLAP workloads.

Clock

CockroachDB employs Hybrid Logical Clocks (HLC) combining physical wall‑time (synchronised via NTP) and a logical component to order events causally. All reads and writes advance the local HLC, and the system enforces a maximum clock offset to maintain performance.

Lock‑Free Distributed Transactions

The database provides lock‑free, serialisable‑snapshot isolation (SSI) transactions, based on research papers such as “Serializable Isolation for Snapshot Databases”. Transactions are recorded with status, key, UUID, timestamp, and priority. Write intents are stored on the originating range and resolved via Raft consensus; a timestamp cache helps detect read‑write, write‑read, and write‑write conflicts.

CDC

CockroachDB supports Change Data Capture (CDC) via changefeed streams that can be consumed by external systems like Kafka, Elasticsearch, ClickHouse, or HBase. Changefeeds are pushed down to each range’s lease‑holder, aggregated, and emitted, allowing linear scalability of CDC workloads.

SQL Interface

Each node acts as a gateway exposing a PostgreSQL‑compatible protocol; CB‑SQL adds MySQL protocol compatibility. SQL statements are compiled into KV operations, with tables having primary keys (implicit if not defined) and column families for storage optimisation. Example key encoding: DB:University(51);Table:student(42);PK:ID(13);Idx:email(14);Idx:name(15) .

Reference

For further reading, see the CockroachDB GitHub repository, Cockroach Labs blog posts on distributed transactions, and several Chinese technical articles linked at the end of the original document.