Accelerating StarRocks Node Decommissioning via tablet_sched_slot_num_per_path Tuning

Problem: Slow Replica Migration During Node Decommissioning

When decommissioning three nodes in a StarRocks storage-compute integrated cluster, each node contained over 1 million tablets. After initiating decommission, only 200,000 replicas had migrated after 24 hours, threatening the decommission schedule.

Root Cause: Default Slot Limits for Replica Scheduling

According to StarRocks documentation, replica repair and balancing tasks are scheduled via slots per disk. By default, each disk is allocated 2 slots for replica repair ( tablet_sched_slot_num_per_path) and 2 separate slots for balancing. A clone task consumes one slot on both source and destination disks. When slots are exhausted, no new tasks are assigned, limiting parallelism.

Solution: Increase tablet_sched_slot_num_per_path During Off-Peak Hours

The author increased the slot count from the default 2 to 32 per disk to boost parallel replica migration.

Step 1: Check Default Value

MySQL [bigdata]> ADMIN SHOW FRONTEND CONFIG LIKE '%tablet_sched_slot_num_per_path%';
+--------------------------------+------------------------------+-------+------+-----------+---------+
| Key                            | AliasNames                   | Value | Type | IsMutable | Comment |
+--------------------------------+------------------------------+-------+------+-----------+---------+
| tablet_sched_slot_num_per_path | [schedule_slot_num_per_path] | 2     | int  | true      |         |
+--------------------------------+------------------------------+-------+------+-----------+---------+
1 row in set (0.00 sec)

Step 2: Increase Slot Count to 32

# Increase to accelerate balancing [monitor disk I/O]
ADMIN SET FRONTEND CONFIG ("tablet_sched_slot_num_per_path" = "32");

Step 3: Monitor Disk I/O

Disk I/O utilization on some nodes reached 100% but did not remain saturated for extended periods.

Result: 4x Faster Decommissioning

Starting from the configuration change, the three nodes completed automatic decommissioning within 8 hours. Each node migrated over 800,000 replicas on average, compared to only 200,000 in the first 24 hours under default settings.

Key Takeaway

For large-scale node decommissioning in StarRocks, temporarily raising tablet_sched_slot_num_per_path during low-traffic windows can dramatically accelerate replica migration. Monitor disk I/O to avoid sustained saturation.