Master Ceph Distributed Storage: Architecture, Benefits, and Step‑by‑Step Deployment Guide

Ceph – Distributed Storage

Ceph is an open‑source, self‑healing, and self‑managing distributed storage system written in C++. It offers high scalability, performance, and reliability, making it a standard storage backend for many cloud platforms such as OpenStack and Kubernetes.

1. Storage Basics

1.1 Single‑node Storage

● DAS (Direct‑Attached Storage) – disks connected directly to a server via IDE, SATA, SCSI, SAS, or USB interfaces, providing block‑level storage.

● NAS (Network‑Attached Storage) – file‑level storage accessed over NFS, CIFS, or FTP; the storage appears as a ready‑made file system.

● SAN (Storage Area Network) – block‑level storage over SCSI, FCSAN, or iSCSI networks.

1.2 Problems of Single‑node Storage

● Limited I/O capacity – even high‑performance SSDs cannot handle massive concurrent accesses.

● Limited capacity – a single disk cannot satisfy large data demands.

● Single point of failure – data loss risk if the node fails.

1.3 Commercial Storage Solutions

Vendors include EMC, NetApp, IBM, Dell, Huawei, Inspur, etc.

1.4 Distributed Storage (Software‑Defined Storage, SDS)

Examples: Ceph, TFS, FastDFS, MooseFS, HDFS, GlusterFS. These systems distribute data across multiple nodes, providing high scalability, performance, and availability.

2. Ceph Overview

2.1 What is Ceph?

Ceph is a C++‑based open‑source distributed storage system with self‑healing and self‑management capabilities, widely adopted by cloud providers.

2.2 Ceph Advantages

● High scalability – decentralized design supports thousands of nodes and scales from TB to EB.

● High reliability – no single point of failure, automatic replication and recovery.

● High performance – uses the CRUSH algorithm for balanced data placement.

● Rich functionality – provides block (RBD), file (CephFS), and object (RADOS Gateway) interfaces.

2.3 Ceph Architecture

Four layers from bottom to top:

RADOS – the reliable, autonomous, distributed object store; consists of OSDs and Monitors.

LIBRADOS – client library providing API access to RADOS.

High‑level interfaces – Object storage (RGW), block storage (RBD), and file system (CephFS).

Application layer – services and clients built on the above interfaces.

2.4 Core Components

OSD (Object Storage Daemon) – handles physical storage, replication, balancing, and recovery. Typically one OSD per disk; at least three OSDs are needed for redundancy.

PG (Placement Group) – virtual grouping of objects; objects are hashed to a PG, then CRUSH maps the PG to OSDs.

Pool – logical namespace containing a set of PGs; provides isolation and fault domains.

#Pool data can be stored as replicated copies (default 3) or using Erasure Code for space efficiency.

Monitor (MON) – stores cluster maps (OSD, Monitor, PG, CRUSH) and handles authentication. A Ceph cluster needs at least three MONs for quorum.

Manager (MGR) – tracks runtime metrics, provides monitoring interfaces (e.g., Prometheus, Zabbix). Requires at least two MGR daemons for HA.

MDS (Metadata Server) – supports CephFS metadata; not required if CephFS is unused.

3. Ceph Data Storage Process

Client obtains the latest Cluster Map from a MON.

Data is split into fixed‑size objects (default 2 MiB or 4 MiB) identified by OID (inode + object number).

OID is hashed; the hash modulo the number of PGs yields a PGID.

CRUSH maps the PGID to specific OSDs, distributing replicas across the cluster.

4. Ceph Version Lifecycle

Ceph releases a new stable version each year (e.g., Nautilus 14.2.0, Mimic 13, etc.). Version numbers follow x.y.z where x denotes the release series, y indicates development (0), release candidate (1), or stable/bug‑fix (2).

5. Ceph Cluster Deployment

5.1 Planning

Recommended hardware and network layout:

10 GbE network for both public and cluster traffic.

Separate public and cluster networks.

MON, MGR, and OSD services on separate hosts (or combined in test labs).

Use SATA disks for OSDs (or better SSDs).

CPU ≥ E5‑2620 v3, memory ≥ 64 GiB.

5.2 Host Configuration Example

Hostname    PublicIP          ClusterIP          Role
admin        192.168.10.120                     admin, client
node01       192.168.10.121    192.168.100.11    mon,mgr,osd(/dev/sdb,/dev/sdc,/dev/sdd)
node02       192.168.10.122    192.168.100.12    mon,mgr,osd(/dev/sdb,/dev/sdc,/dev/sdd)
node03       192.168.10.123    192.168.100.13    mon,osd(/dev/sdb,/dev/sdc,/dev/sdd)
client       192.168.10.124                     client

5.3 Disable SELinux and Firewall

systemctl disable --now firewalld
setenforce 0
sed -i 's/enforcing/disabled/' /etc/selinux/config

5.4 Set Hostnames and /etc/hosts

hostnamectl set-hostname admin
hostnamectl set-hostname node01
hostnamectl set-hostname node02
hostnamectl set-hostname node03
hostnamectl set-hostname client

cat >> /etc/hosts <<EOF
192.168.10.120 admin
192.168.10.121 node01
192.168.10.122 node02
192.168.10.123 node03
192.168.10.124 client
EOF

5.5 Install Common Packages

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum -y install epel-release yum-plugin-priorities yum-utils ntpdate python-setuptools python-pip gcc gcc-c++ autoconf libjpeg libjpeg-devel libpng libpng-devel freetype freetype-devel libxml2 libxml2-devel zlib zlib-devel glibc glibc-devel glib2 glib2-devel bzip2 bzip2-devel zip unzip ncurses ncurses-devel curl curl-devel e2fsprogs e2fsprogs-devel krb5-devel libidn libidn-devel openssl openssh openssl-devel nss_ldap openldap openldap-devel openldap-clients openldap-servers libxslt-devel libevent-devel ntp libtool-ltdl bison libtool vim-enhanced python wget lsof iptraf strace lrzsz kernel-devel pam-devel tcl tk cmake ncurses-devel bison setuptool popt-devel net-snmp screen perl-devel pcre-devel net-snmp screen tcpdump rsync sysstat man iptables sudo libconfig git bind-utils tmux elinks numactl iftop bwm-ng net-tools expect snappy leveldb gdisk python-argparse gperftools-libs conntrack ipset jq libseccomp socat chrony sshpass

5.6 Create Ceph User

useradd cephadm
passwd cephadm
visudo
cephadm ALL=(root) NOPASSWD:ALL

5.7 SSH Key‑Based Authentication

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
ssh-copy-id -o StrictHostKeyChecking=no root@admin
ssh-copy-id -o StrictHostKeyChecking=no root@node01
ssh-copy-id -o StrictHostKeyChecking=no root@node02
ssh-copy-id -o StrictHostKeyChecking=no root@node03

5.8 Time Synchronization

systemctl enable --now chronyd
timedatectl set-ntp true
timedatectl set-timezone Asia/Shanghai
chronyc -a makestep
timedatectl status
timedatectl set-local-rtc 0

5.9 Install Ceph Repository

wget https://download.ceph.com/rpm-nautilus/el7/noarch/ceph-release-1-1.el7.noarch.rpm --no-check-certificate
rpm -ivh ceph-release-1-1.el7.noarch.rpm --force

5.10 Install ceph‑deploy Tool

mkdir -p /etc/ceph
cd /etc/ceph
yum install -y ceph-deploy
ceph-deploy --version

5.11 Generate Initial Configuration

cd /etc/ceph
ceph-deploy new --public-network 192.168.10.0/24 --cluster-network 192.168.100.0/24 node01 node02 node03

This creates ceph.conf and keyring files in /etc/ceph.

5.12 Deploy MON Nodes

cd /etc/ceph
ceph-deploy mon create node01 node02 node03
ceph-deploy --overwrite-conf mon create-initial
ceph-deploy gatherkeys node01

5.13 Distribute Configuration and Admin Keyring

cd /etc/ceph
ceph-deploy --overwrite-conf config push node01 node02 node03
ceph-deploy admin node01 node02 node03

5.14 Deploy OSD Nodes

Ensure disks are unpartitioned, then create OSDs:

# Example for node01 using /dev/sdb
ceph-deploy osd create node01 --data /dev/sdb
# Repeat for other nodes and disks as needed

5.15 Verify OSD Status

ceph osd stat
ceph osd df
ceph -s

5.16 Deploy MGR Nodes

cd /etc/ceph
ceph-deploy mgr create node01 node02

After deployment, check status: ceph -s If a warning about insecure global_id reclaim appears, disable it:

ceph config set mon auth_allow_insecure_global_id_reclaim false

5.17 Enable Dashboard Module

yum install -y ceph-mgr-dashboard
ceph mgr module enable dashboard --force
ceph config set mgr mgr/dashboard/ssl false
ceph config set mgr mgr/dashboard/server_addr 0.0.0.0
ceph config set mgr mgr/dashboard/server_port 8000
ceph mgr services   # shows the dashboard URL

echo "12345678" > dashboard_passwd.txt
ceph dashboard set-login-credentials admin -i dashboard_passwd.txt
# or
ceph dashboard ac-user-create admin administrator -i dashboard_passwd.txt

Access the dashboard at http://<MON_IP>:8000 using the credentials created above.