How Ceph Monitor Uses Paxos to Ensure Consistent Metadata Management

Background

Ceph Monitor is the metadata‑management component of Ceph. Built on an improved Paxos algorithm, it provides external clients with consistent access to and updates of cluster metadata.

Positioning

RADOS is the core of Ceph, offering object, block, and file storage. Managing metadata—mapping keys to data locations—is essential for scalability. Two classic approaches exist:

Centralized metadata management: a single node handles detection, updates, and maintenance. Advantages: simple design, timely updates. Disadvantages: single‑point failure and limited scalability.

Peer‑to‑peer metadata management: all nodes and clients share the load. Advantages: no single‑point failure, horizontal scalability. Disadvantages: slower state propagation, especially in large clusters.

Ceph adopts a hybrid: a central Monitor, but with several enhancements to mitigate the drawbacks of pure centralization.

Ceph’s Enhancements

CRUSH algorithm : translates a key and current cluster state into target OSDs, drastically reducing the amount of metadata the Monitor must handle.

Clustered Monitor : Deploys multiple Monitor nodes to alleviate single‑node bottlenecks.

Intelligent storage nodes : OSDs cache metadata, perform most data accesses directly, handle replication, strong consistency, failure detection, migration, and recovery, thereby offloading work from the Monitor.

Tasks and Data

The Monitor maintains cluster‑wide metadata such as the OSDMap, MonitorMap, and other essential information. Typical interactions include:

Clients obtain the current cluster state and CRUSH map from a Monitor on first access.

OSDs report failures to a Monitor.

When an OSD joins or recovers, it contacts a Monitor to receive the latest state.

Implementation Overview

The Monitor’s internal structure consists of three layers: PaxosService , Paxos , and LevelDB . The Paxos layer provides a consistent key‑value interface for all metadata types, using the Paxos algorithm to achieve consensus across nodes.

Bootstrap Phase

When a node starts or after a majority‑failure event, it enters the Bootstrap process. The node probes peers, determines data freshness, and performs full synchronization with nodes that are far behind. This guarantees communication with a majority of nodes and keeps data gaps small.

Recovery Phase

The elected leader collects the current commit position from the quorum and updates the entire cluster, ensuring the cluster state is consistent and the cluster becomes available.

Read/Write Phase

The leader completes writes using a two‑phase commit and updates followers’ leases. All followers holding a valid lease can serve read requests, distributing read load across the Monitor set.

Consistency and Paxos Design Choices

Bootstrap to simplify quorum : Quorum is fixed after leader election; any node join/leave triggers a new Bootstrap, simplifying Paxos at the cost of occasional overhead.

Leader selection by IP : Leader election and data update are separated into distinct stages.

Leader‑only proposals : Only the leader can initiate a propose, and only one value at a time.

Leases : Distribute read pressure across all Monitor nodes, enhancing horizontal scalability for read‑heavy workloads.

Aggregated updates : Non‑MonitorMap writes are batched before being committed, reducing update pressure thanks to intelligent OSDs that reconcile inconsistencies locally.

Conclusion

The article highlighted Ceph Monitor’s Paxos‑based consistency mechanisms, its architectural choices, and the three main operational phases. A deeper dive into the implementation details will be covered in a future article.