Comprehensive Ceph Operations Handbook: Common Tasks, Troubleshooting, and Advanced Topics

Overview

Distributed storage is essential for building elastic data platforms in enterprises. Ceph is the leading open‑source solution, offering high scalability, multi‑protocol support, and a mature ecosystem.

Part 1 – Common Ceph Operations

Start, stop, and restart Ceph daemons (MON, OSD, MDS, RGW).

Monitor cluster health via ceph health and the dashboard.

Manage users and authentication keys with ceph auth.

Add or remove MON nodes and OSD devices, including disk preparation, ceph-volume commands, and OSD deployment.

Create, delete, and modify storage pools; set size, min_size, pg_num, and pgp_num.

Edit cluster configuration using ceph config set and persist changes in ceph.conf.

Update the CRUSH map: export with ceph osd getcrushmap -o crushmap.bin, edit (e.g., crushtool –decompile), and inject the new map.

Change monitor IP addresses by updating mon_host in the configuration and restarting the affected MONs.

Part 2 – Fault Handling

The manual groups typical failure scenarios and provides step‑by‑step diagnostics:

OSD down or out – check ceph osd tree, examine logs, verify network connectivity, and use ceph osd down / ceph osd in to recover.

MON quorum loss – inspect ceph mon dump, ensure quorum IPs are reachable, and restart missing MONs.

PG stuck in inactive/undersized – run ceph pg dump, rebalance with ceph osd reweight or adjust pg_num.

Data placement issues – verify CRUSH rules, bucket hierarchy, and device‑class assignments.

Authentication failures – confirm client keyrings and caps, regenerate keys if needed.

Each case includes log locations, common error messages, and the exact commands required to isolate and resolve the problem.

Part 3 – Advanced Ceph Topics

Advanced sections cover deeper deployment and performance tuning, such as:

Optimizing OSD journal or Bluestore settings (e.g., bluestore_block_wal_size, bluestore_cache_size).

Fine‑tuning CRUSH rules for heterogeneous hardware.

Configuring multi‑site replication with Ceph RBD mirroring or CephFS snapshots.

Integrating Ceph with container orchestration platforms and service meshes, highlighting the trend of service‑mesh adoption in cloud‑native environments.