How to Separate SSD and SATA OSDs in Ceph Using Custom CRUSH Rules

Background

In a Ceph cluster that contains both SSD and SATA drives, the default CRUSH placement distributes placement groups (PGs) evenly across all OSDs, which can waste SATA capacity and degrade performance. By customizing the CRUSH map, high‑IO data can be stored on SSD OSDs while less‑critical data uses SATA OSDs, improving performance and reducing cost.

Get the current CRUSH map and decompile it

ceph osd getcrushmap -o crushmapdump
crushtool -d crushmapdump -o crushmapdump-decompiled

Edit the decompiled file and, after the root default section, add two new buckets: one for SSD OSDs (osd.0, osd.2, osd.4) and one for SATA OSDs (osd.1, osd.3, osd.5).

root ssd {
    id -5
    alg straw
    hash 0
    item osd.0 weight 0.010
    item osd.2 weight 0.010
    item osd.4 weight 0.010
}

root sata {
    id -6
    alg straw
    hash 0
    item osd.1 weight 0.010
    item osd.3 weight 0.010
    item osd.5 weight 0.010
}

Create CRUSH rules

Each pool uses its own CRUSH rule set. Define a rule for SSD‑backed pools and another for SATA‑backed pools.

rule ssd-pool {
    ruleset 1
    type replicated
    min_size 1
    max_size 10
    step take ssd   # use SSD bucket
    step chooseleaf firstn 0 type osd
    step emit
}
rule sata-pool {
    ruleset 2
    type replicated
    min_size 1
    max_size 10
    step take sata   # use SATA bucket
    step chooseleaf firstn 0 type osd
    step emit
}

Compile and inject the new CRUSH map

crushtool -c crushmapdump-decompiled -o crushmapdump-compiled
ceph osd setcrushmap -i crushmapdump-compiled

Add the following line to ceph.conf to prevent Ceph from resetting the map on restart:

osd_crush_update_on_start=false

After applying the map, view the OSD tree to confirm the bucket layout (images omitted for brevity).

Create and verify the SSD pool

ceph osd pool create ssd-pool 8 8

Set the pool’s CRUSH rule to the SSD rule (rule ID 1) and verify that the pool uses the SSD bucket.

ceph osd pool set ssd-pool crush_rule ssd-pool

Similarly, create a SATA pool and assign it rule ID 2, confirming that it uses the SATA bucket.

Write data to both pools and validate placement

rados -p <pool_name> put <object_name> <file_name>

Inspect the objects to ensure they reside on the correct OSD sets: SSD OSDs are [0, 2, 4] and SATA OSDs are [1, 3, 5]; the verification screenshots show the expected distribution.