How to Deploy a Multi‑Node Ceph Cluster on CentOS 7 – Step‑by‑Step Guide

Basic Configuration

Three CentOS 7.9 machines (node1, node2, node3) require the following steps on each host.

Configure hosts resolution

cat >> /etc/hosts <<EOF
192.168.2.16 node1
192.168.2.19 node2
192.168.2.18 node3
EOF

Disable firewall and SELinux

systemctl stop firewalld && systemctl disable firewalld
setenforce 0 && sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

Set hostnames

hostnamectl set-hostname node1
hostnamectl set-hostname node2
hostnamectl set-hostname node3

Synchronize time

systemctl restart chronyd.service && systemctl enable chronyd.service

Install Packages with yum

Install yum‑plugin‑priorities

yum install yum-plugin-priorities

Install dependencies

yum install snappy leveldb gdisk python-argparse gperftools-libs epel-release

Add Ceph repository (Aliyun mirror)

vim /etc/yum.repos.d/ceph.repo
[ceph]
name=ceph
baseurl=http://mirrors.aliyun.com/ceph/rpm-15.2.8/el7/x86_64/
gpgcheck=0
[ceph-noarch]
name=cephnoarch
baseurl=http://mirrors.aliyun.com/ceph/rpm-15.2.8/el7/noarch/
gpgcheck=0

Install Ceph

yum install ceph -y

Deploy monitor nodes

Each Ceph cluster needs at least one monitor. The guide creates three monitors on node1, node2, and node3.

Create monitor on node1

Generate a unique FSID: uuidgen Add the FSID to the Ceph configuration:

vim /etc/ceph/ceph.repo
[global]
fsid=9c079a1f-6fc2-4c59-bd4d-e8bc232d33a4

Create monitor keyring:

ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'

Create admin keyring:

ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin \
  --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'

Create bootstrap‑OSD keyring:

ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd \
  --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'

Import keyrings:

ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring

Set ownership: chown ceph:ceph /tmp/ceph.mon.keyring Create monitor map:

monmaptool --create --add node1 192.168.2.16 \
  --add node2 192.168.2.19 \
  --add node3 192.168.2.18 \
  --fsid 9c079a1f-6fc2-4c59-bd4d-e8bc232d33a4 /tmp/monmap

Initialize monitor on node1:

ceph-mon --mkfs -i node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring

Create data directory and start service:

mkdir -p /var/lib/ceph/mon/ceph-node1
chmod 777 -R /var/lib/ceph/mon/ceph-node1
systemctl start ceph-mon@node1 && systemctl enable ceph-mon@node1

Repeat the same steps on node2 and node3 (create directories, import keyring, run ceph-mon --mkfs, and start the service).

Create manager (MGR) daemons

On each node:

sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-`hostname -s`
cd /var/lib/ceph/mgr/ceph-`hostname -s`
ceph auth get-or-create mgr.`hostname -s` mon 'allow profile mgr' osd 'allow *' mds 'allow *' > keyring
systemctl enable ceph-mgr@`hostname -s` && systemctl start ceph-mgr@`hostname -s`

If the manager fails due to missing Python module, install it:

pip3 install pecan werkzeug

Deploy OSDs

Install the ceph-volume utility and create OSDs using LVM: ceph-volume lvm create --data /dev/sdb The process can be split into preparation and activation:

ceph-volume lvm prepare --data /dev/sdb
ceph-volume lvm list   # shows OSD ID and FSID
ceph-volume lvm activate {ID} {FSID}

Copy the bootstrap‑OSD keyring to the other nodes and adjust ownership:

scp /var/lib/ceph/bootstrap-osd/ceph.keyring node2:/var/lib/ceph/bootstrap-osd/ceph.keyring
scp /var/lib/ceph/bootstrap-osd/ceph.keyring node3:/var/lib/ceph/bootstrap-osd/ceph.keyring
chown ceph.ceph /var/lib/ceph/bootstrap-osd/ceph.keyring

Repeat the ceph-volume lvm create command on node2 and node3.

Add MDS daemons

Create data directories and keyrings for each MDS:

mkdir -p /var/lib/ceph/mds/ceph-`hostname -s`
chown -R ceph.ceph /var/lib/ceph/mds/ceph-`hostname -s`
ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-`hostname -s`/keyring \
  --gen-key -n mds.`hostname -s`
ceph auth add mds.`hostname -s` osd "allow rwx" mds "allow" mon "allow profile mds" \
  -i /var/lib/ceph/mds/ceph-`hostname -s`/keyring
chown ceph:ceph /var/lib/ceph/mds/ceph-`hostname -s`/keyring

Update ceph.conf:

cat >> /etc/ceph/ceph.conf <<EOF
[mds.node1]
host = node1

[mds.node2]
host = node2

[mds.node3]
host = node3
EOF

Start MDS services:

systemctl enable ceph-mds@`hostname -s` && systemctl start ceph-mds@`hostname -s`

Deploy RADOS Gateway (RGW)

Install the RGW package: yum install ceph-radosgw -y Create the required pools (example creates each with 8 placement groups):

ceph osd pool create .rgw 8 8
ceph osd pool create .rgw.root 8 8
ceph osd pool create .rgw.control 8 8
ceph osd pool create .rgw.gc 8 8
ceph osd pool create .rgw.buckets 8 8
ceph osd pool create .rgw.buckets.index 8 8
ceph osd pool create .rgw.buckets.extra 8 8
ceph osd pool create .log 8 8
ceph osd pool create .intent-log 8 8
ceph osd pool create .usage 8 8
ceph osd pool create .users 8 8
ceph osd pool create .users.email 8 8
ceph osd pool create .users.swift 8 8
ceph osd pool create .users.uid 8 8

If pool creation fails because of the default mon_max_pg_per_osd limit, increase it in ceph.conf and restart the monitors:

vim /etc/ceph/ceph.conf
[global]
mon_max_pg_per_osd = 1000
systemctl restart ceph-mon@`hostname -s`

Create RGW client keyrings and grant permissions:

ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
chown ceph:ceph /etc/ceph/ceph.client.radosgw.keyring
ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.rgw.node1 --gen-key
ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.rgw.node2 --gen-key
ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.rgw.node3 --gen-key
ceph-authtool -n client.rgw.node1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
ceph-authtool -n client.rgw.node2 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
ceph-authtool -n client.rgw.node3 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.rgw.node1 -i /etc/ceph/ceph.client.radosgw.keyring
ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.rgw.node2 -i /etc/ceph/ceph.client.radosgw.keyring
ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.rgw.node3 -i /etc/ceph/ceph.client.radosgw.keyring

Append RGW sections to ceph.conf:

cat >> /etc/ceph/ceph.conf <<EOF
[client.rgw.node1]
host=node1
keyring=/etc/ceph/ceph.client.radosgw.keyring
log file=/var/log/radosgw/client.radosgw.gateway.log
rgw_frontends = civetweb port=8080

[client.rgw.node2]
host=node2
keyring=/etc/ceph/ceph.client.radosgw.keyring
log file=/var/log/radosgw/client.radosgw.gateway.log
rgw_frontends = civetweb port=8080

[client.rgw.node3]
host=node3
keyring=/etc/ceph/ceph.client.radosgw.keyring
log file=/var/log/radosgw/client.radosgw.gateway.log
rgw_frontends = civetweb port=8080
EOF

Create log directory and set ownership:

mkdir -p /var/log/radosgw
chown ceph:ceph /var/log/radosgw

Copy the keyring and configuration to node2 and node3, adjust ownership, and start the RGW daemons:

scp /etc/ceph/ceph.client.radosgw.keyring node2:/etc/ceph/ceph.client.radosgw.keyring
scp /etc/ceph/ceph.client.radosgw.keyring node3:/etc/ceph/ceph.client.radosgw.keyring
scp /etc/ceph/ceph.conf node2:/etc/ceph/ceph.conf
scp /etc/ceph/ceph.conf node3:/etc/ceph/ceph.conf
chown ceph:ceph /etc/ceph/ceph.client.radosgw.keyring
systemctl start ceph-radosgw@rgw.`hostname -s` && systemctl enable ceph-radosgw@rgw.`hostname -s`

Verify cluster health with ceph health detail and test the RGW endpoint using curl (e.g., curl http://node1:8080).