Insights from the 58 Group Technical Salon: Distributed KV Storage Systems Cellar and WTable

The first 58 Group technical salon, held on October 12, 2018 at the Beijing headquarters, featured presentations by Meituan‑Dianping and 58 infrastructure teams on distributed KV storage technologies, specifically Cellar and WTable.

Cellar Overview

Cellar evolved from Taobao's open‑source Tair KV store, inheriting Tair's multi‑engine support (Memcache, LevelDB, RocksDB) and asynchronous replication for eventual consistency. It routes keys by hashing them into a fixed number of buckets, with routing tables maintained by a ConfigServer that clients query once and cache.

Key improvements over Tair include decentralizing the ConfigServer via an ObServer layer, using Zookeeper for ConfigServer master‑slave election, and implementing intelligent data‑migration throttling, backup‑request handling for network jitter, and request classification to isolate slow operations.

WTable Overview

WTable is 58's home‑grown distributed KV store built on RocksDB, with configuration consistency and leader election handled by etcd. Data is partitioned into 2048 slots, each assigned to a Group; a Group contains identical replicas, with one primary elected via etcd.

Clients obtain a Proxy list from a NameCenter service, hash keys to slots, map slots to Groups via etcd, and then route requests to the appropriate storage nodes.

Scaling optimizations include slot‑based expansion with a two‑phase migration (full data copy followed by incremental sync) to minimize service impact, and a fast backup‑node addition method that creates a RocksDB checkpoint and copies compressed SST files directly.

Comparative Summary

Both Cellar and WTable handle massive daily traffic (over a trillion accesses) and are highly reliable, yet they differ in data organization, central‑node usage, and operational trade‑offs. Cellar distributes buckets across all nodes, offering flexible scaling and resilience to node failures, while WTable relies on strict Group replication, simplifying scheduling but limiting the number of nodes that can participate in recovery.

The discussion highlighted the importance of avoiding single points of failure, optimizing data migration to reduce resource contention, and ensuring service availability under network jitter or slow queries.

Next Salon Preview

The next technical salon will continue exploring advanced infrastructure topics.