How Baidu Cloud Accelerates Data Lakes with Compute‑Storage Separation

Big Data Technology Evolution

Big‑data architectures progressed from single‑machine processing in the 1960s‑70s to parallel MPP systems, then to Hadoop in 2006, spawning engines such as MapReduce, Hive, Spark, and Flink. Since 2013, cloud‑native concepts like data lakes and compute‑storage separation have become prominent.

Overview of Baidu Intelligent Cloud Big Data Solution

The solution stacks object storage (BOS) as the data‑lake layer, two compute engines (managed BMR platform and Doris data‑warehouse), and the EasyDAP analytics platform for integration, governance, development, analysis, and service, all backed by unified metadata management.

Advantages of Compute‑Storage Separation

Elasticity: Compute and storage can be scaled independently, avoiding resource waste when workloads change.

Cost Efficiency: Object storage enables tiered cold‑storage (up to 1/8 of standard storage cost) and pay‑as‑you‑go compute, reducing the need for always‑on HDFS clusters.

Lower Operations Overhead: Cloud providers manage object‑storage operations, eliminating the heavy maintenance burden of large‑scale HDFS clusters.

Challenges of Compute‑Storage Separation

Flat Namespace in Object Storage: Renaming deep directories requires listing and moving every file, leading to latency proportional to file count.

I/O Path Length: Accessing data via BOS traverses multiple layers (load balancer → Webservice → metadata → storage), roughly doubling the hop count compared to HDFS.

Bandwidth Consumption: Separate compute and storage clusters generate massive cross‑cluster traffic (often >10 TB), stressing network links.

Lack of Data Locality: Compute nodes cannot directly sense data placement, causing uniform access patterns and higher I/O overhead.

Data Lake Acceleration Solutions

1. Native Hierarchical Namespace

BOS introduces a hierarchical namespace that transforms flat directories into a tree structure. Directory rename operations become constant‑time (milliseconds) regardless of the number of files, and the API remains compatible with existing code. A one‑click switch between flat and hierarchical namespaces is provided.

2. RapidFS Cache Acceleration

RapidFS offers two modes:

Cache Mode: Metadata is cached locally, mirroring BOS metadata for faster reads.

Block Mode: Metadata is persisted in RapidFS, decoupling it from BOS metadata and enabling ultra‑low‑latency VPC‑internal access.

Data‑side caching uses intelligent file‑type detection to pre‑fetch hot index blocks into a distributed cache, dramatically improving query performance.

Performance Evaluation

Two benchmark sets were executed on 10 TB‑scale TPC‑DS workloads using SparkSQL and Hive‑Tez:

Comparing flat vs. hierarchical namespace showed ~10 % average speedup, with metadata‑intensive queries (q5, q45, q62) gaining >40 %.

RapidFS vs. direct BOS access yielded >15 % average improvement, with I/O‑heavy queries (q9, q28, q88) achieving >30 % gains.

All results were normalized; lower latency indicates better performance.

Best‑Practice Deployment Patterns

Pattern 1: Hierarchical Namespace + RapidFS Cache Mode – ideal for multiple compute clusters sharing the same BOS bucket.

Pattern 2: Flat Namespace + RapidFS Block Mode – suited for one‑to‑one relationships between a compute cluster (e.g., BMR) and a dedicated BOS bucket.

Pattern 3: Flat Namespace + RapidFS Cache Mode – adds a transparent cache layer to existing workloads without code changes, useful when migration cost is high.

Typical VPC‑internal metadata latency is 300‑400 µs; accessing BOS metadata over the network adds 1‑2 ms, making RapidFS Block Mode optimal for low‑latency needs.