How to Benchmark Ceph Cluster Performance: A Step‑by‑Step Guide

Before deploying a Ceph cluster in production, you should benchmark it to obtain rough results for read, write, latency, and other workloads. First establish a hardware baseline for disks and network, then perform disk performance tests using dd with flags that bypass cache.

Test Single Disk Write Performance

Clear page cache echo 3 > /proc/sys/vm/drop_caches Write a 10 GB file named zero filled with zeros to the Ceph OSD directory:

dd if=/dev/zero of=/var/lib/ceph/osd/ceph-0/zero bs=10G count=1

Repeat the test and average the results; note that kernel cache was not bypassed, so numbers may be high.

for i in `mount | grep osd | awk '{print $3}'`; do (dd if=/dev/zero of=$i/zero bs=10G count=1 &) ; done

OSD Single Disk Read Performance

Clear page cache

echo 3 > /proc/sys/vm/drop_caches

dd if=/var/lib/ceph/osd/ceph-0/deleteme of=/dev/null bs=10G count=1 iflag=direct

for i in `mount | grep osd | awk '{print $3}'`; do (dd if=$i/zero of=/dev/null bs=10G count=1 &) ; done

Network Baseline Performance

Test the network between Ceph OSD nodes using iperf. Install iperf, start a server on node 1 (port 6900), and run a client on node 2.

apt install iperf

rados bench Benchmark

Ceph includes the rados bench tool for pool performance testing. Example commands:

rados bench -p libvirt-pool 10 write --no-cleanup

rados bench -p libvirt-pool 10 seq

rados bench -p libvirt-pool 10 rand

Syntax:

rados bench -p <pool_name> <seconds> <write|seq|rand> -b <block size> -t <threads> --no-cleanup

RADOS load‑gen

The rados load-gen tool generates load on a Ceph cluster for stress testing.

rados -p libvirt-pool load-gen --num-objects 200 --min-object-size 4M --max-object-size 8M --max-ops 10 --read-percent 0 --min-op-len 1M --max-op-len 4M --target-throughput 2G --run-length 20

Block Device Benchmark

Use the RBD bench-write command to benchmark a Ceph RADOS Block Device.

rbd create libvirt-pool/289 --size 10240 --image-feature layering
rbd info -p libvirt-pool --image 289
rbd map libvirt-pool/289
rbd showmapped
mkfs.xfs /dev/rbd2
mkdir -p /mnt/289
mount /dev/rbd2 /mnt/289
df -h /mnt/289
rbd bench-write libvirt-pool/289 --io-total 5368709200

Syntax of rbd bench-write:

rbd bench-write <RBD image name> --io-size 4M --io-threads 16 --io-total 1024M --io-pattern seq|rand

Using fio to Benchmark Ceph RBD

Install fio and create a configuration file write.fio:

apt install fio -y

[write-4M]
ioengine=rbd
direct=1
size=5g
lockmem=1G
runtime=30
group_reporting
numjobs=1
iodepth=32
pool=libvirt-pool
clientname=admin
rbdname=289
rw=write
bs=4M
filename=/dev/rbd0

Run the test:

fio write.fio