How to Deploy a Ceph Distributed Storage Cluster from Scratch

Ceph Overview

Ceph is a petabyte‑scale distributed storage system that provides object, block, and file interfaces. It consists of Monitor nodes, MDS nodes for CephFS, and OSD daemons that store data.

Cluster Components

Ceph OSD : stores data, handles recovery, rebalancing and reports status to monitors. At least two OSD daemons on separate physical servers are required.

Monitor : maintains cluster maps (monmap, osdmap, pgmap). A minimum of three monitors is recommended for high availability.

MDS : provides metadata services for CephFS.

Manager : tracks runtime metrics and exposes the dashboard and REST API.

Preparation of Nodes

Three monitor nodes (192.168.85.128‑131) and two OSD nodes (192.168.85.133‑134) are used. Each OSD node must have two disks and be provisioned with 4 GiB RAM, 4 vCPU and 60 GB storage.

Hostname and Hosts Configuration

hostnamectl set-hostname monitor1

hostnamectl set-hostname monitor2

hostnamectl set-hostname monitor3

hostnamectl set-hostname osd1

hostnamectl set-hostname osd2

192.168.85.128 monitor1
192.168.85.130 monitor2
192.168.85.131 monitor3
192.168.85.133 osd1
192.168.85.134 osd2

Password‑less SSH

ssh-keygen -t rsa   # accept defaults
ssh-copy-id -i .ssh/id_rsa.pub monitor1
ssh-copy-id -i .ssh/id_rsa.pub monitor2
ssh-copy-id -i .ssh/id_rsa.pub monitor3
ssh-copy-id -i .ssh/id_rsa.pub osd1
ssh-copy-id -i .ssh/id_rsa.pub osd2

Base Packages and Firewall

yum -y install wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel vim ncurses-devel autoconf automake zlib-devel python-devel

yum install deltarpm -y

systemctl stop firewalld.service && systemctl disable firewalld.service

yum install iptables-services -y
service iptables stop && systemctl disable iptables

Time Synchronization

# on master1
ntpdate cn.pool.ntp.org
systemctl start ntpd && systemctl enable ntpd

# on other nodes
ntpdate monitor1
# cron job (run hourly)
* */1 * * * /usr/sbin/ntpdate monitor1

Install ceph‑deploy and Ceph Repositories

yum install -y yum-utils && sudo yum-config-manager --add-repo https://dl.fedoraproject.org/pub/epel/7/x86_64/ && sudo yum install --nogpgcheck -y epel-release && sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 && sudo rm -rf /etc/yum.repos.d/dl.fedoraproject.org*

# Ceph repo (Jewel on CentOS 7)
[Ceph]
name=Ceph packages for $basearch
baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/
enabled=1
gpgcheck=0
type=rpm-md
gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc
priority=1
[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch/
enabled=1
gpgcheck=0
type=rpm-md
gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc
priority=1

yum update -y
yum install ceph-deploy ceph -y   # on master1
yum install yum-plugin-priorities -y   # on master1
yum install ceph -y   # on other nodes

Create Monitor Cluster

mkdir /root/ceph-deploy && cd /root/ceph-deploy
ceph-deploy new monitor1 monitor2 monitor3

Generated files include ceph.conf, ceph-deploy-ceph.log, and ceph.mon.keyring.

# Example ceph.conf (global section)
[global]
fsid = dc97ea52-e32c-404d-9bbc-9d0a90e39e02
mon_initial_members = monitor1, monitor2, monitor3
mon_host = 192.168.85.128,192.168.85.129,192.168.85.130
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd_pool_default_size = 2   # adjust for two OSDs

Deploy Ceph Packages

ceph-deploy install monitor1 monitor2 monitor3 osd1 osd2

Initialize Monitors and Distribute Keys

ceph-deploy mon create-initial

Key files such as ceph.bootstrap-mds.keyring, ceph.bootstrap-osd.keyring, and ceph.client.admin.keyring appear in the deployment directory.

ceph-deploy admin monitor1 monitor2 monitor3 osd1 osd2

chmod +r /etc/ceph/ceph.client.admin.keyring

Configure OSDs

On each OSD node:

mkdir -p /data/osd1
chmod 777 /data/osd1
chown -R ceph:ceph /data/osd1

Prepare and activate OSDs from the master:

ceph-deploy osd prepare osd1:/data/osd1 osd2:/data/osd2
ceph-deploy osd activate osd1:/data/osd1 osd2:/data/osd2

Verify Cluster Health

ceph health   # should show HEALTH_OK

Create a Test Pool and RBD Image

ceph osd pool create testpool 256
rbd create testpool/myrbd --size 10240
rbd feature disable testpool/myrbd object-map fast-diff deep-flatten
rbd map testpool/myrbd   # shows /dev/rbd0

Mount the RBD Block Device

mkdir /mnt/firstrbd
mkfs.xfs /dev/rbd0
mount /dev/rbd0 /mnt/firstrbd

Common Troubleshooting

Too many PGs per OSD

When ceph -s reports HEALTH_WARN: too many PGs per OSD (320 > max 300), increase the warning threshold:

ceph --show-config | grep mon_pg_warn_max_per_osd   # default 300
# edit /etc/ceph/ceph.conf on all monitors
mon_pg_warn_max_per_osd = 600
systemctl restart ceph-mon@monitor1
systemctl restart ceph-mon@monitor2
systemctl restart ceph-mon@monitor3

Monitor service fails to start

If systemd reports “start request repeated too quickly”, edit /usr/lib/systemd/system/[email protected] and comment out StartLimitInterval=30min, then reload the daemon:

systemctl daemon-reload
systemctl restart ceph-mon@monitor1
systemctl restart ceph-mon@monitor2
systemctl restart ceph-mon@monitor3

Full Cluster Re‑deployment

To start over, purge all components from the master:

cd /root/ceph-deploy
ceph-deploy purge monitor1 monitor2 monitor3 osd1 osd2
ceph-deploy purgedata monitor1 monitor2 monitor3 osd1 osd2
rm ceph.*