Ctrip Cloud Platform Overview and Ceph Deployment Cases

Cloud computing has become a key driver of innovation for enterprises worldwide, and Ctrip has leveraged it by building a private cloud platform based on OpenStack that provides IaaS for development, testing, and production environments across multiple data centers.

Ceph is introduced as a unified distributed storage system offering object, block, and file interfaces. Its core, RADOS, underpins the cluster composed of MON, OSD, and MDS daemons, while the librados library enables interaction with the storage system. Ceph’s services include the object gateway (RGW) and block device interface (RBD).

Ceph RGW acts as a RESTful gateway compatible with S3 and Swift APIs. Requests flow from clients through Apache FastCGI to RGW, which translates them into RADOS operations via librados, handling both data and extensive metadata such as bucket, object, and user information.

Ceph RBD provides block storage through two implementations: a kernel‑mode driver (kRBD) that communicates directly with RADOS via a net module, and a user‑space library (librbd) that offers caching (RBD cache) for performance. The kernel driver lacks the cache but relies on the page cache.

Ctrip’s deployment uses Ceph version 0.94.2 (H) on CentOS 6.4, with four clusters located in Fuqian, Ouyang, Jinqiao, and Nantong. Each of the three production servers runs 12 OSDs, one MON, and an RGW instance, achieving HA and load balancing through DNS round‑robin.

Two primary use cases are described: (1) storing hotel image feature vectors—millions of small 20‑30 KB files—where Ceph RGW efficiently handles space usage and metadata; (2) supporting cross‑data‑center continuous delivery, initially exploring Crushmap replication and radosgw‑agent synchronization, ultimately leading to a custom COS platform built on Celery, RabbitMQ, and HA components for reliable data and metadata replication.

The future roadmap includes a unified Ceph‑Manager for centralized cluster management and additional components such as Data Sync, QA databases, picture feature services, and deeper integration with OpenStack.

Author: Liu Jun