Mastering Ceph: From Overview to Cluster Deployment and Management

1. Ceph Overview

Ceph is a decentralized distributed storage system offering high performance, reliability and scalability. Originating from a 2004 research project, it is LGPL‑licensed and now supported by many cloud platforms such as OpenStack, CloudStack and Hadoop.

1.2 Features

High performance – client and server communicate directly without proxies, using CRUSH algorithm for balanced data distribution across thousands of nodes.

High availability – multiple replicas, no single point of failure, automatic detection and recovery.

High scalability – fully parallel design, linear growth with added nodes.

Rich scenarios – supports object, block and file storage interfaces.

2. Ceph Architecture

The core storage layer is RADOS (Reliable Autonomic Distributed Object Store). Above it sits the librados library, then high‑level services such as RADOS Gateway (RGW), RBD and CephFS, and finally application‑specific clients.

2.1 Terminology

OSD – object storage daemon handling data placement, replication and recovery.

Monitor – maintains cluster maps and health status.

PG – placement group that aggregates objects for mapping.

MDS – metadata server for CephFS.

RADOS – underlying object store.

Librados – API library for direct object access.

RBD – block device service.

RGW – S3/Swift compatible gateway.

3. Ceph Cluster Deployment

Three CentOS 7 nodes (192.168.116.141‑143) are prepared, hostnames are set, /etc/hosts edited, NTP synchronized, firewall disabled and SELinux set to disabled.

vi /etc/hostname

192.168.116.141 CENTOS7-1
192.168.116.142 CENTOS7-2
192.168.116.143 CENTOS7-3

Ceph repository is configured using the Tsinghua mirror, then ceph and ceph‑deploy are installed. yum update && yum -y install ceph ceph-deploy Password‑less SSH is set up for a dedicated ceph_user account.

useradd -d /home/ceph_user -m ceph_user
passwd ceph_user
ssh-keygen
ssh-copy-id ceph_user@CENTOS7-1

Cluster is created with ceph-deploy new, monitors initialized, configuration edited, and daemons installed.

ceph-deploy new CENTOS7-1 CENTOS7-2 CENTOS7-3
ceph-deploy mon create-initial
ceph-deploy install CENTOS7-1 CENTOS7-2 CENTOS7-3
ceph-deploy admin CENTOS7-1 CENTOS7-2 CENTOS7-3

The dashboard module is enabled, a self‑signed certificate generated, and the web UI configured on port 18843.

ceph mgr module enable dashboard
ceph dashboard create-self-signed-cert
ceph config set mgr mgr/dashboard/server_addr 192.168.116.141
ceph config set mgr mgr/dashboard/server_port 18843
ceph config set mgr mgr/dashboard/ssl false
ceph dashboard set-login-credentials admin admin

3.6 CephFS Creation

Two pools are created for data and metadata, then a CephFS named fs_test is defined and mounted via ceph-fuse.

ceph osd pool create cephfs_data 128
ceph osd pool create cephfs_metadata 64
ceph fs new fs_test cephfs_metadata cephfs_data
ceph-fuse -k /etc/ceph/ceph.client.admin.keyring -m 192.168.116.141:6789 /usr/local/cephfs_directory

3.7 Java Client (RADOS)

The rados‑java source is built with Maven, the resulting rados-0.7.0.jar is placed in the JDK extensions, and a simple Java program connects to the cluster.

git clone https://github.com/ceph/rados-java.git
mvn install -Dmaven.test.skip=true
java CephClient