How Baidu’s BTS Powers High‑Performance, Multi‑Model NoSQL at Scale

BTS is Baidu Intelligent Cloud's semi‑structured storage product that supports core Baidu services (search, Apollo, feed, system monitoring, etc.) and offers high‑performance, low‑cost NoSQL table storage to both internal and external users.

1. BTS Overview

BTS serves a wide range of scenarios, including horizontal business (distributed storage, aggregation, high‑performance retrieval), vertical industry (Internet, advertising, IoT, big data, time‑series), and integrated solutions (big‑data analytics, monitoring). It provides multiple APIs, SDKs, and a visual web console for rapid integration, uses batch writes, concurrent reads, multi‑level cache acceleration, hot‑standby replicas, real‑time failover, and a recycle‑bin for high availability, achieving 99.9% service availability and 99.99999999% data reliability.

The product has a 12‑year history divided into three generations:

1.0 (2011): First‑generation distributed table, supporting billions of entries for internal services.

2.0 (2015): Added FreeSchema, compute‑storage separation, and sparse tables for advertising workloads.

3.0 (2018): Cloud service launch, supporting feed recommendation, AI, and scaling to millions of billions of entries; introduced a time‑series engine and began multi‑model engine reconstruction.

2. System Architecture

The architecture consists of three layers: access, engine, and storage.

Storage Layer manages data blocks with a compute‑storage separation design, offering modular storage, multiple media types, and compression formats for lifecycle management.

Engine Layer provides core data processing, stateless modular components, and intelligent scheduling based on multi‑dimensional policies. It includes a multi‑level cache and an automatic slice‑merge mechanism that can predict workload changes for proactive scheduling.

Access Layer supports HBase and Influx interfaces, with more APIs under development.

2.1 Core Technology Design

NoSQL databases aim for high performance, low cost, high availability, and scalability.

2.1.1 Single‑Node Engine Read/Write Path

Write path: client → Redo‑Log → in‑memory table → dump → compressed storage (supports multi‑level compression).

Read path: client → cache lookup → if miss, fetch from storage → merge with in‑memory data → return to client.

2.1.2 Performance Optimization & Cost Management

GroupCommit merges small writes to boost throughput, especially on HDD.

Multiple compaction strategies reduce I/O amplification.

Concurrent I/O + prefetch improves read throughput.

Configurable multi‑level cache (memory, SSD) accelerates hot data access.

Various compression types (three‑replica, Snappy, EC) and tiered storage (capacity vs. performance) lower storage cost.

2.2 High‑Availability Architecture

BTS employs a HA framework with control and worker nodes. Worker nodes are hot‑standby and synchronize data via Redo‑Log. The control node handles fast failover, achieving sub‑hundred‑millisecond switchover and seamless node upgrades.

This design provides multi‑level fault tolerance, tenant isolation, end‑to‑end data verification, real‑time monitoring, and overall high availability.

3. Use Cases

3.1 Autonomous Driving

Apollo's autonomous vehicle training requires multi‑source data (position, radar, image, infrared) at TB‑scale per car per day, with high real‑time demands. BTS addresses this by partitioning data by vehicle and sensor, dispersing writes, and using auto‑split/merge storage units to avoid hotspots. Data cooling, quarterly table partitioning, and tiered storage reduce costs while delivering PB‑scale storage and GB‑scale throughput.

3.2 System Monitoring

Monitoring generates massive small‑value metrics from servers, VMs, and containers. BTS separates offline and online traffic, slicing offline tasks into tiny chunks to allow online tasks to preempt resources. Multi‑level caching keeps hot data in fast storage, while cold data resides on HDD, achieving >80% cache hit rate, >50% cost reduction, and supporting tens of trillions of points per day with zero data loss.

4. Future Outlook

Beyond wide‑table and time‑series support, BTS will add document and search multimodal capabilities, enabling unified analysis of multiple data modalities in a single task.

Ba​idu Intelligent Cloud's database portfolio (RDS, NoSQL, cloud‑native DB, OLAP) offers a unified architecture across public cloud, private cloud (ABC Stack), edge (BEC), and on‑premise clusters (LCC), delivering consistent product capabilities for diverse customer needs.

以上是今天分享的全部内容。