How Baidu’s Canghai Storage Tackles Massive Data Challenges in the Cloud

1. Four storage challenges in the ABC era

We call the current period the ABC era: A for Artificial Intelligence, B for Big Data, C for the era where everything can move to the cloud.

Storage systems now face four key challenges:

Massive scale : Data volumes are exploding with video, audio, etc., requiring cloud providers to handle petabyte‑scale growth.

Cost‑performance : Customers need to store growing data without proportional cost increases.

Stability : Distributed systems must guarantee high reliability and disaster‑recovery capabilities.

Diversity : Varied business scenarios such as big‑data analytics, AI training, and hybrid‑cloud deployments demand flexible storage solutions.

2. Overview of Baidu Canghai Storage product system

Baidu Canghai Storage supports core Baidu services such as Search, Netdisk, Tieba, Baijiahao, Maps, and AI workloads.

The product matrix includes Object Storage (BOS), Block Storage (CDS), File Storage (CFS), Parallel File Storage (PFS), and specialized offerings like RapidFS for data‑lake acceleration, CloudFlow for data migration, Moonlight Box for physical data transfer, and storage gateways for hybrid‑cloud integration.

3. How Canghai Storage solves the four challenges

3.1 Full‑scene data migration and cloud‑on‑boarding

Data sources include on‑premise IDC and other clouds (AWS, Tencent, Alibaba). Migration options are disk‑array hybrid cloud, Moonlight Box (physical transfer), dedicated lines, and CloudFlow for cross‑cloud sync.

CloudFlow provides a visual, one‑click interface for source and destination configuration, supporting incremental sync and mirror‑back for continuous availability.

3.2 Intelligent lifecycle management

BOS offers tiered storage (Standard, Infrequent, Cold, Archive) and lifecycle rules that automatically migrate data based on age, reducing costs dramatically (archive storage costs ~18% of standard).

Up‑floating rules allow hot promotion of cold data when access frequency spikes.

3.3 Multi‑level disaster recovery and reliability

BOS guarantees 12‑nine (99.9999999999%) reliability through erasure‑coding across multiple AZs, and availability of 99.95% (single AZ) to 99.99% (multi‑AZ), with real‑world availability around 99.9995%.

Disaster‑recovery mechanisms include physical‑machine failover, multi‑AZ replication, cross‑region backup, and data‑mirror back‑source.

3.4 Integrated multi‑product workflow

Solutions cover big‑data lake acceleration (RapidFS + BOS), hybrid‑cloud storage via storage gateway BSG, and AI high‑performance computing using BOS with Parallel File System (PFS) and POSIX‑compatible acceleration.

These capabilities enable customers such as iQIYI to lower storage costs while maintaining global content distribution.