Understanding Ceph’s Scrub Mechanism: How Data Integrity Is Verified and Its Current Challenges

Ceph Foundation Overview

On November 12, 2018, the Linux Foundation announced the creation of the Ceph Foundation in Berlin to provide a neutral organization that supports the Ceph open‑source project. The foundation is governed by the Linux Foundation and its board consists of senior members, regular members, associate members, and representatives of the Ceph leadership team.

The board’s responsibilities include approving the annual budget for Ceph, establishing special committees to address current project needs, coordinating outreach, holding regular meetings to discuss foundation activities and Ceph’s strategic direction, and voting on decisions presented to the board.

The board does not perform technical governance of Ceph; development and engineering are managed through the usual open‑source processes under the supervision of the Ceph leadership team.

Membership and Adoption

Founding top‑tier members include Canonical, China Mobile, DigitalOcean, Intel, Red Hat, SUSE, Western Digital, and others. Regular and associate members span a wide range of organizations such as ARM, CERN, various universities, and cloud providers.

Ceph is now one of the most popular distributed software‑defined storage systems, serving financial institutions, cloud service providers, academic and government bodies, telecom operators, automotive manufacturers, and software solution vendors.

Numerous commercial products in China are built on Ceph, including H3C’s ONEstor, ZTE’s CloveStorage, and Inspur’s AS13000 series.

Scrub Mechanism Overview

Scrub is a periodic data‑verification process that runs on each placement group (PG) to check consistency between replicas. Two modes exist:

Scrub : scans only metadata, fast.

Deep‑Scrub : scans both metadata and actual stored data, slower and more I/O‑intensive.

Default schedules are daily to weekly for Scrub (depending on cluster load) and weekly for Deep‑Scrub, running across the full 0‑24 hour window.

Scrub Workflow

Each PG’s master OSD initiates the Scrub process. Only a subset of objects in the PG is examined per run, and those objects are locked against modification during verification.

The OSD extracts a ScrubMap containing each object’s size, extended‑attribute keys, and historical version information. This map is compared with the maps from other replicas.

If inconsistencies are found, the mismatched objects are collected and reported to the Monitor.

Administrators can manually trigger repair with ceph pgrepair <pg_id>, which copies the master’s copy to the replicas. Current repair assumes the master holds the correct data.

Deep‑Scrub performs a full object‑by‑object comparison, which can impose a heavy I/O load and is therefore rarely used in production.

Current Limitations

Two main issues are identified:

After detecting inconsistent objects, Ceph lacks an automatic correction strategy; it relies on manual repair and does not automatically reconcile minority replicas with the majority.

The Scrub mechanism does not provide end‑to‑end data integrity guarantees, meaning applications may read corrupted data before the Scrub cycle detects the problem.

To address the first issue, Ceph has a Blueprint that will enable majority‑vote‑based repair when users invoke the repair command. For the second issue, traditional end‑to‑end verification (e.g., adding checksums or ECC at the storage device level) is not feasible for Ceph because it operates above the filesystem and object layer, and such checks would significantly impact performance.

Conclusion

The Scrub mechanism is essential for maintaining data consistency in Ceph clusters, but it currently depends on manual intervention for correction and cannot guarantee immediate end‑to‑end integrity. Ongoing development aims to improve automatic repair and explore complementary verification techniques.