How UCloud Revamped US3 Metadata Service for 80% Cost Savings and Faster Performance

1. Introduction

Object storage is a flat‑hierarchy data storage method commonly used in the cloud, where each object has a unique identifier and optional metadata. In 2006 Amazon launched AWS S3, introducing object storage to cloud computing, and UCloud later developed its own US3 service.

2. Challenges of the US3 Metadata Service

The US3 metadata service now handles exabyte‑scale storage with billions of daily index accesses, upload requests, and delete operations, creating intense index‑update pressure and large‑scale list scans.

The original architecture relied on MongoDB as the underlying store with external routing and monitoring services. While initially scalable, explosive growth in customers and data exposed performance, scalability, and cost limitations.

3. Previous US3 Metadata Architecture

Metadata was stored in a chained MongoDB cluster; when one cluster reached write limits, another was added downstream. Queries passed through a DB‑Gateway that translated JSON to BSON and routed requests across multiple MongoDB clusters. Deletions required broadcasting to all involved clusters.

Directly sharding MongoDB was avoided due to migration‑induced latency. As data grew, MongoDB read‑write performance degraded, and list‑service scans became a bottleneck, prompting the separation of a dedicated list service that synchronized data from MongoDB.

Performance pain points: chained architecture caused write underutilization and read amplification.

Data consistency issues: deletions across multiple clusters could leave orphaned documents.

List‑service synchronization delays prevented real‑time file visibility.

Operational pain points: poor scalability required manual addition of MongoDB clusters.

Cost pain points: additional machines for list service increased expenses.

4. New US3 Metadata Service Architecture

The redesigned system consolidates three components: a highly compatible DB‑Gateway, a highly available distributed KV store (UKV) that separates compute and storage, and a customized RocksDB variant (URocksDB). Metadata now resides in UKV, offering stronger disaster recovery, faster hotspot splitting, scalable performance, erasure‑coded storage, and heterogeneous media support.

Improvements

Performance boost : reduced metadata deletion latency and read amplification.

Zero list‑service delay : clients see uploaded data instantly, eliminating delay alerts.

Cost reduction : metadata service cost lowered by 80%.

Simpler operations : compute nodes achieve disaster recovery without data migration; hotspot nodes split automatically, removing manual scaling.

5. Core Changes of the New Architecture

DB‑Gateway is a stateless process that maintains compatibility with the old service, provides UKV sharding routing, and aggregates former list‑service functions, eliminating the need for separate list‑service machines.

UKV is UCloud’s self‑developed distributed KV storage system that separates compute and storage. Its storage layer, Manul, offers automatic data balancing, heterogeneous media, erasure coding, high performance, and high availability.

UKV includes cluster management, fast backup, and a data model tailored for US3 metadata, delivering superior performance in UCloud’s daily workloads. It runs on a customized RocksDB variant (URocksDB) and leverages Manul for hot‑standby replication and rapid hotspot node splitting.

Further details on UKV’s design and unique features will be shared in future posts.