Unlock Scalable, Reliable Storage: A Complete Guide to Deploying Ceph

Ceph – Distributed Storage

1. Storage Basics

1.1 Single‑node Storage

● DAS (Direct‑Attached Storage) – disks connected directly to a server via IDE, SATA, SCSI, SAS, or USB, providing block‑level storage.

● NAS (Network‑Attached Storage) – file‑level storage accessed over the network via NFS, CIFS, FTP; appears as a ready‑made file system to clients.

● SAN (Storage Area Network) – block‑level storage over networks using SCSI, FCSAN, iSCSI protocols.

Issues with single‑node storage include insufficient I/O capacity, limited space, and single‑point failures.

2. Ceph Overview

2.1 Introduction

Ceph is an open‑source, self‑healing, self‑managing distributed storage system written in C++. It offers high scalability, performance, and reliability, and is widely supported by cloud platforms such as RedHat, OpenStack, and Kubernetes.

2.2 Advantages

High scalability – decentralized design supports thousands of nodes from TB to EB.

High reliability – no single‑point failure, multiple data replicas, automatic recovery.

High performance – CRUSH algorithm provides balanced data distribution and parallelism.

Rich functionality – unified system with block (RBD), file (CephFS), and object (RadosGW) interfaces.

2.3 Architecture

Ceph consists of four layers from bottom to top:

RADOS – core object store built from OSD and Monitor daemons, providing infinite scalability.

LIBRADOS – client library exposing APIs for higher‑level services.

High‑level interfaces – RGW (object), RBD (block), CephFS (POSIX file system).

Application layer – hosts client applications, VMs, etc.

Key components:

OSD (Object Storage Daemon) – handles data storage, replication, recovery, and heartbeat.

PG (Placement Group) – virtual grouping of objects, similar to database indexes, mapped to OSDs via CRUSH.

Pool – logical namespace containing multiple PGs.

Monitor – maintains cluster maps and authentication.

Manager – tracks runtime metrics and provides monitoring interfaces.

MDS (Metadata Server) – supports CephFS metadata operations.

3. Distributed Storage Types

● Block storage – raw devices for VM disks, logs, etc.

● File storage – NFS‑like shared storage for directories and logs.

● Object storage – API‑driven storage (e.g., OSS) suitable for images, videos.

4. Ceph Version Lifecycle

Ceph releases a new stable version each year (e.g., Nautilus 14.2.0, Mimic 13). Version format x.y.z where x is the release series, y indicates development (0), candidate (1), or stable (2).

5. Deploying a Ceph Cluster with ceph‑deploy

5.1 Environment Planning

admin   192.168.10.120  admin,client</code><code>node01 192.168.10.121  mon,mgr,osd(/dev/sdb,/dev/sdc,/dev/sdd)</code><code>node02 192.168.10.122  mon,mgr,osd(/dev/sdb,/dev/sdc,/dev/sdd)</code><code>node03 192.168.10.123  mon,osd(/dev/sdb,/dev/sdc,/dev/sdd)</code><code>client 192.168.10.124  client

5.2 Preparing Hosts

useradd cephadm</code><code>passwd cephadm</code><code>visudo</code><code>cephadm ALL=(root) NOPASSWD:ALL

5.3 Disable SELinux and Firewall

systemctl disable --now firewalld</code><code>setenforce 0</code><code>sed -i 's/enforcing/disabled/' /etc/selinux/config

5.4 Set Hostnames and /etc/hosts

hostnamectl set-hostname admin</code><code>hostnamectl set-hostname node01</code><code>hostnamectl set-hostname node02</code><code>hostnamectl set-hostname node03</code><code>hostnamectl set-hostname client</code><code>cat >> /etc/hosts <<EOF</code><code>192.168.10.120 admin</code><code>192.168.10.121 node01</code><code>192.168.10.122 node02</code><code>192.168.10.123 node03</code><code>192.168.10.124 client</code><code>EOF

5.5 Install Dependencies

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo</code><code>yum -y install epel-release yum-plugin-priorities ntpdate python-setuptools python-pip gcc gcc-c++ autoconf libjpeg-devel libpng-devel freetype-devel libxml2-devel zlib-devel glibc-devel glib2-devel bzip2-devel zip unzip ncurses-devel curl-devel e2fsprogs-devel krb5-devel libidn-devel openssl-devel nss_ldap-devel openldap-devel libxslt-devel libevent-devel ntp libtool bison vim-enhanced wget lsof iptraf strace lrzsz kernel-devel kernel-headers pam-devel tcl tk cmake ncurses-devel popt-devel net-snmp-devel screen perl-devel pcre-devel net-snmp-devel tcpdump rsync sysstat iptables sudo libconfig git bind-utils tmux elinks numactl iftop bwm-ng

5.6 Install ceph‑deploy

cd /etc/ceph</code><code>yum install -y ceph-deploy</code><code>ceph-deploy --version

5.7 Create Cluster Configuration

cd /etc/ceph</code><code>ceph-deploy new --public-network 192.168.10.0/24 --cluster-network 192.168.100.0/24 node01 node02 node03

5.8 Deploy Monitors

cd /etc/ceph</code><code>ceph-deploy mon create node01 node02 node03</code><code>ceph-deploy --overwrite-conf mon create-initial

5.9 Distribute Admin Key

ceph-deploy admin node01 node02 node03

5.10 Deploy OSDs

# Zap old disks (optional)</code><code>ceph-deploy disk zap node01 /dev/sdb</code><code>ceph-deploy disk zap node02 /dev/sdb</code><code>ceph-deploy disk zap node03 /dev/sdb</code><code># Create OSDs</code><code>ceph-deploy --overwrite-conf osd create node01 --data /dev/sdb</code><code>ceph-deploy --overwrite-conf osd create node02 --data /dev/sdb</code><code>ceph-deploy --overwrite-conf osd create node03 --data /dev/sdb

5.11 Deploy Managers

ceph-deploy mgr create node01 node02

5.12 Enable Dashboard

yum install -y ceph-mgr-dashboard</code><code>ceph mgr module enable dashboard --force</code><code>ceph config set mgr mgr/dashboard/ssl false</code><code>ceph config set mgr mgr/dashboard/server_addr 0.0.0.0</code><code>ceph config set mgr mgr/dashboard/server_port 8000</code><code>echo "12345678" > dashboard_passwd.txt</code><code>ceph dashboard set-login-credentials admin -i dashboard_passwd.txt

5.13 Verify Cluster Health

ceph -s

Typical health warnings such as mons are allowing insecure global_id reclaim can be resolved with:

ceph config set mon auth_allow_insecure_global_id_reclaim false

6. Additional Operations

Commands for checking OSD status, expanding the cluster, and monitoring include ceph osd stat, ceph osd df, ceph quorum_status --format json-pretty, and ceph osd tree. The guide also covers time synchronization, SSH key distribution, and disabling unnecessary services.