Mastering Ceph: From Block, File to Object Storage and Step-by-Step Deployment

Block Storage: Typical Devices, Disk Arrays, Hard Drives

Purpose : Map raw disk space to hosts; disk arrays perform RAID or LVM to create logical volumes that appear as separate disks to the OS, improving I/O efficiency through parallel access.

Advantages : Increased read/write efficiency by aggregating multiple disks.

Disadvantages : Treated as local disks, cannot be shared across hosts.

File Storage

Typical Devices : FTP, NFS servers Advantages : Enables file sharing, overcoming block storage limitations.

Disadvantages : Lower read/write and transfer speeds.

Use Cases : Logs, email, hierarchical file storage.

Note : NFS and NAS are common file storage solutions.

Object Storage

Definition :

Object storage, also known as object‑based storage, describes a method for handling discrete units called objects.

It is often likened to valet parking: the user receives a receipt (object ID) without needing to know the physical location of the car (data).

Advantages : Combines high‑speed block I/O with file‑sharing capabilities.

Data Types : Unstructured data such as images, audio/video, documents.

Typical Scenario : Write‑once, read‑many workloads.

Distributed Storage

Distributed storage spreads data across multiple independent devices, avoiding the bottleneck of a single centralized server and improving reliability, availability, and scalability.

Key benefits include high availability with flexible replica counts, strong consistency, no single point of failure, high scalability, and rich feature sets.

Ceph Overview

Ceph is a unified distributed storage system offering block, object, and file interfaces. A Ceph cluster requires at least one monitor (MON) and two OSD daemons; the file system client also needs a metadata server (MDS).

Ceph Advantages

1. CRUSH Algorithm : Replaces centralized metadata lookup with a scalable, fault‑domain‑aware hashing mechanism for data placement.

2. High Availability : Administrators can define replica counts and placement rules; Ceph ensures strong consistency and automatic recovery.

3. High Scalability : No single master node; performance scales linearly with added disks.

4. Rich Features : Supports object, block, and file interfaces simultaneously; widely used as the sole backend for OpenStack in many Chinese cloud environments.

Ceph Components

OSDs : Store data, handle replication, recovery, back‑fill, and rebalancing; report heartbeats to monitors.

Monitors : Track cluster state, maintain maps of OSDs, placement groups, and CRUSH.

MDS : Metadata server for the Ceph file system, enabling POSIX operations without burdening the storage cluster.

Deployment Steps

1. Prepare three servers (admin, node1, node2) and disable firewalls and SELinux.

2. Configure /etc/hosts for all nodes.

3. Set up password‑less SSH between nodes.

4. Install required packages (wget, curl, net‑tools, bash‑completion) and add Ceph yum repositories.

5. Synchronize time using chrony on both server and client.

6. Install Docker and pull the Ceph daemon image.

7. Start the monitor container on the admin node:

docker run -d --net=host --name=mon \
  -v /etc/ceph:/etc/ceph -v /var/lib/ceph:/var/lib/ceph \
  -e MON_IP=192.168.106.100 -e CEPH_PUBLIC_NETWORK=192.168.106.0/24 \
  ceph/daemon:v3.0.5-stable-3.0-luminous-centos-7-x86_64 mon

8. Adjust OSD configuration for non‑XFS filesystems if needed.

9. Copy Ceph configuration files to the other nodes.

10. Launch monitor containers on node1 and node2 with their respective IPs.

11. Prepare and mount OSD disks (e.g., mkfs.xfs /dev/sdb, mount /dev/sdb /osd0).

12. Start OSD containers on each node.

13. Deploy a manager (mgr) container on node1.

14. Create a storage pool (e.g., docker exec mon ceph osd pool create rbd 64).

15. Configure the CRUSH map to assign OSDs to the default root.

16. Verify the cluster status with docker exec mon ceph -s.

17. Test the deployment by creating an RBD image, inspecting it, and listing objects in the pool.