How to Deploy Ceph Storage for OpenStack: A Complete Step‑by‑Step Guide

Introduction

Ceph is an open‑source distributed storage system offering high reliability, high performance, and linear scalability. OpenStack is a cloud platform that can use Ceph as its backend storage to simplify deployment and reduce operational complexity.

Why Use Ceph with OpenStack

Integrating Ceph with OpenStack provides shared storage for all compute nodes, enabling instant VM migration, zero‑copy cloning via COW, and thin provisioning that allocates space on demand.

Machine Planning and Network Configuration

Three node types are required: controller , compute , and ceph_admin . Their network interfaces are configured as follows:

ens33: floating IP 192.168.199.115 (controller)
ens37: internal network 192.168.184.135 (controller)

ens33: floating IP 192.168.199.220 (compute)
ens37: internal network 192.168.184.136 (compute)

ens33: floating IP 192.168.199.211 (admin)
ens37: internal network 192.168.184.134 (admin)

Host Preparation

Set hostnames with hostnamectl set-hostname <name>.

Disable NetworkManager:

systemctl disable NetworkManager && systemctl stop NetworkManager

.

Disable firewalld: systemctl disable firewalld && systemctl stop firewalld.

Update /etc/hosts on all nodes:

192.168.184.135 controller
192.168.184.136 compute
192.168.184.134 admin

Synchronize time: ntpdate time2.aliyun.com.

Configure password‑less SSH between all nodes using ssh-keygen -t rsa and ssh-copy-id for each pair.

Ceph Repository and Installation

Add the Ceph yum repository:

[Ceph]
name=Ceph packages for $basearch
baseurl=http://mirrors.163.com/ceph/rpm-kraken/el7/$basearch
enabled=1
priority=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.163.com/ceph/rpm-kraken/el7/noarch
enabled=1
priority=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-source]
name=Ceph source packages
baseurl=http://mirrors.163.com/ceph/rpm-kraken/el7/SRPMS
enabled=0
priority=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

Refresh the cache with yum makecache, then install ceph-deploy and create a working directory:

mkdir -p /etc/ceph && cd /etc/ceph
yum -y install ceph-deploy

Cluster Creation

Deploy Ceph on all nodes: ceph-deploy install admin controller compute Initialize the cluster configuration: ceph-deploy new admin controller compute The command generates ceph.conf containing:

[global]
fsid = 37516893-56ee-44b8-807f-04e7b253e1e1
mon_initial_members = admin, controller, compute
mon_host = 192.168.184.134,192.168.184.135,192.168.184.136
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

Create monitors, gather keys, and deploy OSDs:

ceph-deploy mon create admin controller compute
ceph-deploy gatherkeys admin
ceph-deploy osd create admin:sdb controller:sdb compute:sdb

Distribute the configuration and set keyring permissions:

ceph-deploy admin admin controller compute
chmod +r /etc/ceph/ceph.client.admin.keyring

Verify cluster health:

ceph -s

Ceph Pool Management

Create pools for OpenStack services:

ceph osd pool create volumes 128   # Cinder
ceph osd pool create vms 128       # Nova
ceph osd pool create images 128    # Glance

List pools to confirm:

ceph osd lspools

Integration with OpenStack Services

Glance

Enable RBD backend in /etc/glance/glance-api.conf:

stores = rbd
rbd_store_pool = images
rbd_store_user = glance
rbd_store_ceph_conf = /etc/ceph/ceph.conf
rbd_store_chunk_size = 8
default_store = rbd

Restart Glance services and verify images appear with rbd ls images.

Cinder

Install python-rbd on the controller, then configure /etc/cinder/cinder.conf:

glance_api_version = 2
enabled_backends = ceph,lvm

[ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid = 502804dd-8504-405a-b7a1-33a52ec3a77c

Restart Cinder volume service, create the Ceph‑backed volume type, and create a test volume:

cinder type-create ceph
cinder type-key ceph set volume_backend_name=ceph
cinder create --volume-type ceph --name ceph-volume1 2

Nova

On each compute node, edit /etc/nova/nova.conf to use RBD images:

[libvirt]
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid = 9a144709-dc79-48d9-9008-0f299c76d6d1
disk_cachemodes="network=writeback"
inject_password=false
inject_key=false
inject_partition=-2
hw_disk_discard=unmap

Restart the compute service and upload a raw image (converted from QCOW2) to Glance for use by Nova.

Cleanup

To remove the Ceph cluster:

ceph-deploy purge admin controller compute
ceph-deploy purgedata admin controller compute
ceph-deploy forgetkeys

All steps above provide a reproducible method to deploy Ceph, create necessary pools, and integrate it with OpenStack’s core services.