Master Ceph on Linux: Complete Guide to Deploying and Managing a Production-Ready Cluster

Linux Distributed Storage Solution: Complete Ceph Cluster Deployment and Operations Guide

Introduction: Why Choose Ceph?

As a senior operations engineer, I have witnessed many enterprises struggle with storage architecture selection. Traditional NAS/SAN solutions are expensive and lack scalability, while cloud storage introduces vendor lock‑in risks. After deep diving into Ceph, I realized it represents the future of software‑defined storage.

In this article I share, without reservation, my full experience of deploying and operating Ceph clusters in production, including pitfalls and optimization tricks that official documentation often omits.

What Is Ceph? More Than Just Distributed Storage

Ceph is not merely a distributed storage system; it is a unified storage platform that simultaneously provides:

Object Storage (RADOS Gateway) : S3/Swift compatible API

Block Storage (RBD) : High‑performance disks for VMs

File System (CephFS) : POSIX‑compatible distributed file system

This "three‑in‑one" architecture makes Ceph an ideal choice for enterprise storage consolidation.

Core Advantages of Ceph

No Single Point of Failure : Truly decentralized architecture

Dynamic Scaling : PB‑level expansion with online scaling

Self‑Healing : Automatic data balancing and recovery

Open‑Source Ecosystem : Avoid vendor lock‑in, strong community support

Production‑Grade Ceph Cluster Architecture Design

Hardware Recommendations

Based on multiple production deployments, the following configuration is recommended:

Monitor Nodes (minimum 3, odd number)

CPU: 4 cores or more
Memory: 8 GB or more
Disk: SSD 100 GB (system disk)
Network: Dual 10 GbE NICs (redundant)

OSD Nodes (suggested start with 6)

CPU: 1 core per OSD
Memory: 4 GB per OSD (BlueStore)
Disk: Enterprise SSD or high‑rpm HDDs
Network: Dual 10 GbE NICs (public + cluster network)

MGR Nodes (minimum 2)

CPU: 2 cores
Memory: 4 GB
Disk: System disk is sufficient

Network Architecture Design

Key point often overlooked by engineers:

# Public (client access)
10.0.1.0/24

# Cluster network (data replication and heartbeat)
10.0.2.0/24

Core Principle : Separate client traffic from internal cluster traffic to avoid network congestion affecting cluster stability.

Hands‑On Ceph Cluster Deployment

Environment Preparation

# 1. System requirements (example: CentOS 8)
cat /etc/os-release

# 2. Time synchronization (critical!)
systemctl enable --now chronyd
chrony sources -v

# 3. Firewall configuration
firewall-cmd --zone=public --add-port=6789/tcp --permanent
firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent
firewall-cmd --reload

# 4. SELinux settings
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

Install cephadm Tool

# Install official package manager
curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
chmod +x cephadm
./cephadm add-repo --release octopus
./cephadm install

Initialize Cluster

# 1. Bootstrap first Monitor node
cephadm bootstrap --mon-ip 10.0.1.10 --cluster-network 10.0.2.0/24

# 2. Install Ceph CLI tools
cephadm install ceph-common

# 3. Verify cluster status
ceph status

Successful bootstrap shows output similar to:

cluster:
  id: a7f64266-0894-4f1e-a635-d0aeaca0e993
  health: HEALTH_OK

Add OSD Nodes

# 1. Copy SSH keys to all nodes
ssh-copy-id root@node2
ssh-copy-id root@node3

# 2. Add hosts to the cluster
ceph orch host add node2 10.0.1.11
ceph orch host add node3 10.0.1.12

# 3. List available disks
ceph orch device ls

# 4. Add OSDs
ceph orch daemon add osd node2:/dev/sdb
ceph orch daemon add osd node2:/dev/sdc
ceph orch daemon add osd node3:/dev/sdb
ceph orch daemon add osd node3:/dev/sdc

Configure Storage Pools

# 1. Create replicated pool (3 replicas)
ceph osd pool create mypool 128 128 replicated

# 2. Set application type
ceph osd pool application enable mypool rbd

# 3. Set CRUSH rule for rack‑level fault tolerance
ceph osd crush rule create-replicated rack_rule default rack
ceph osd pool set mypool crush_rule rack_rule

Production Operations Practices

Performance Monitoring and Tuning

Key Monitoring Metrics

# 1. Overall cluster health
ceph health detail

# 2. Storage usage
ceph df

# 3. OSD performance stats
ceph osd perf

# 4. Slow request monitoring
ceph osd slow-requests

# 5. PG status distribution
ceph pg stat

Performance Tuning Parameters

Create an optimized configuration file /etc/ceph/ceph.conf:

[global]
# Network tuning
ms_bind_port_max = 7300
ms_bind_port_min = 6800

# OSD tuning
osd_max_write_size = 512
osd_client_message_size_cap = 2147483648
osd_deep_scrub_interval = 2419200
osd_scrub_max_interval = 604800

# BlueStore tuning
bluestore_cache_size_hdd = 4294967296
bluestore_cache_size_ssd = 8589934592

# Recovery control
osd_recovery_max_active = 5
osd_max_backfills = 2
osd_recovery_op_priority = 2

Troubleshooting Cases

Case 1: OSD Down

# 1. View detailed health
ceph health detail
# 2. Locate down OSD
ceph osd tree | grep down
# 3. Check OSD logs
journalctl -u ceph-osd@3 -f
# 4. Restart OSD
systemctl restart ceph-osd@3
# 5. If hardware failure, mark out and replace
ceph osd out 3

Case 2: PG Inconsistency

# Find inconsistent PGs
ceph pg dump | grep inconsistent
# Repair specific PG
ceph pg repair 2.3f
# Deep scrub
ceph pg deep-scrub 2.3f

Case 3: Disk Space Exhaustion

# Check usage
ceph df detail
# Identify most used pool
ceph osd pool ls detail
# Temporarily raise alert thresholds
ceph config set global mon_osd_full_ratio 0.95
ceph config set global mon_osd_backfillfull_ratio 0.90
ceph config set global mon_osd_nearfull_ratio 0.85
# Long‑term solution: add OSDs or delete data
ceph orch daemon add osd node4:/dev/sdb

Capacity Planning and Expansion

Capacity Calculation

Usable Capacity = Raw Capacity × (1 - Replication Factor/Replication Factor) × (1 - Reserved Ratio)
# Example: 100 TB raw, 3‑replica, 10% reserve
# Usable = 100 TB × (1 - 3/3) × (1 - 0.1) = 30 TB

Smooth Expansion Process

# 1. Pre‑add settings
ceph config set global osd_max_backfills 1
ceph config set global osd_recovery_max_active 1

# 2. Add OSDs one by one
ceph orch daemon add osd node5:/dev/sdb
# Wait for data rebalance
ceph -w

# 3. Restore defaults
ceph config rm global osd_max_backfills
ceph config rm global osd_recovery_max_active

Backup and Disaster Recovery

RBD Snapshot Backup

# Create snapshot
rbd snap create mypool/myimage@snapshot1
# Export snapshot
rbd export mypool/myimage@snapshot1 /backup/myimage.snapshot1
# Cross‑cluster mirroring
rbd mirror pool enable mypool image
rbd mirror image enable mypool/myimage

Cluster‑Level Backup

# Export configuration
ceph config dump > /backup/ceph-config.dump
# Backup CRUSH map
ceph osd getcrushmap -o /backup/crushmap.bin
# Backup monitor data
ceph-mon --extract-monmap /backup/monmap

Security Hardening

# Enable authentication
ceph config set mon auth_cluster_required cephx
ceph config set mon auth_service_required cephx
ceph config set mon auth_client_required cephx

# Create dedicated user
ceph auth get-or-create client.backup mon 'allow r' osd 'allow rwx pool=mypool'

# Enable network encryption
ceph config set global ms_cluster_mode secure
ceph config set global ms_service_mode secure

Automation Script Example (Health Check)

#!/bin/bash
# ceph-health-check.sh
LOG_FILE="/var/log/ceph-health.log"
ALERT_EMAIL="[email protected]"

check_health() {
    HEALTH=$(ceph health --format json | jq -r '.status')
    if [ "$HEALTH" != "HEALTH_OK" ]; then
        echo "$(date): Cluster health is $HEALTH" >> $LOG_FILE
        ceph health detail >> $LOG_FILE
        echo "Ceph cluster health issue detected" | mail -s "Ceph Alert" $ALERT_EMAIL
    fi
}

check_capacity() {
    USAGE=$(ceph df --format json | jq -r '.stats.total_used_ratio')
    THRESHOLD=0.80
    if (( $(echo "$USAGE > $THRESHOLD" | bc -l) )); then
        echo "$(date): Storage usage is ${USAGE}" >> $LOG_FILE
        echo "Storage capacity warning" | mail -s "Ceph Capacity Alert" $ALERT_EMAIL
    fi
}

main() {
    check_health
    check_capacity
}

main

Summary and Outlook

By following this in‑depth guide you should now have a solid grasp of Ceph cluster deployment and operation in production environments. Ceph is not just a storage solution; it is a foundational component for enterprise digital transformation. Mastering Ceph operations positions you at the technical forefront of distributed storage.

Key Takeaways :

Architecture Design : Proper hardware selection and network planning are prerequisites for success.

Monitoring & Operations : Establish a comprehensive monitoring system to prevent issues before they arise.

Performance Tuning : Adjust parameters based on workload characteristics to achieve optimal performance.

Fault Handling : Rapid identification and resolution of problems is a core competency.

As cloud‑native technologies evolve, Ceph’s role in containerized and micro‑service architectures will continue to grow. Owning Ceph operational skills will give you a strategic advantage in the distributed storage domain.