CynosDB Storage Cluster Manager (SCM): Architecture, Data Model, and Scheduling Mechanisms

CynosDB is a distributed relational database system built on top of CynosFS. To maximize storage utilization and balance resource competition, it relies on a Storage Cluster Manager (SCM) that uses Etcd for metadata storage and Raft for leader election. SCM exposes HTTP APIs for querying CynosFS status and handles two types of scheduling: Pool scheduling and Segment Group (SG) scheduling.

Key Components and Terminology

• DBEngine : Database engine supporting one primary and multiple replicas. • DB Cluster Manager (DCM) : Manages HA for DB clusters. • Storage Service / Storage Node (SN) : Logical service providing storage; each SN is a process handling logs, block replay, multi‑version reads, and WAL backup. • Segment (Seg) : Smallest unit (≈10 GB) of block and log data, replicated via Raft, forming a Segment Group (SG). • Pool : Logical block device composed of multiple SGs; a Pool can contain many SGs. • Storage Cluster Manager (SCM) : Manages Storage Service, maintains Pool‑SG relationships, and schedules SG members.

SCM Data Model

The SCM data model consists of four information categories: API Info (RESTful and gRPC interfaces), System Info (global config, member URLs, etc.), Schedule Info (jobs and status for Pools and SGs), and Store Info (metadata and runtime statistics of Storage Services).

I/O Model

SCM Master interacts with Storage Service (resource scheduling, heartbeats), Distributed File System (provides Pool/SG info for routing), DCM (creates Pools, queries SG status), and SCM Slave (synchronizes data via Etcd for high availability).

Scheduling Principles

Scheduling is driven by heartbeats (Protobuf V3 over gRPC). SNs send heartbeats; SCM processes them, generates commands, and returns responses. Commands include segment allocation, SG creation, leader switches, and replica migrations.

Pool Scheduling

Pool scheduler polls all Pools every 5 seconds, handling initialization, expansion, contraction, and deletion. Pool states: UNINITIALIZE, NORMAL, DISABLE, DELETE. Scheduling states: INITIATING, EXPANDING, EXPANDED, IDLE. Expansion adds a new SG (e.g., SG1) to the Pool, persists metadata to Etcd, and updates status accordingly. Contraction reverses this process, removing SGs from highest ID downwards. Pool release frees all SGs.

Segment Group (SG) Scheduling

Each Pool has an SG scheduler that polls every second. SG states: UNINITIALIZE, NORMAL, DISABLE. Segment states include UNINITIALIZE, INITIATING, ALLOCATING, ALLOCATED, BOOTSTRAPPING, ADDING, ADDED, REMOVING, RELEASING. SG scheduling involves SN heartbeats (SCM receives statistics, issues segment/SG operations) and SG heartbeats (SCM updates SG metadata and issues replica adjustments).

Packing Algorithm

The algorithm selects Storage Nodes for SG replicas through four steps: filtering, filtering, scoring, and comparison.

1. Filter : Based on Pool tags (e.g., REGION, ZONE), select candidate stores. 2. Filter : Exclude stores that are offline, overloaded, or already part of the SG. 3. Score : Compute a score for each candidate by comparing its location to existing SG stores (region, zone, rack, host). 4. Compare : If scores tie, choose the store with fewer existing Segments.

This process ensures balanced replica placement across regions, zones, racks, and hosts.