Automated Resource Balancing Scheduling in Redis Management Platform
Gotu’s Redis management platform now runs an unattended 5 AM scheduler that automatically selects and migrates nodes based on priority rules, verifies master‑slave synchronization, and rebalances host memory to stay below safe thresholds, improving utilization, reducing DBA effort, and supporting future auto‑deployment and recovery.
Gotu's Redis management platform oversees hundreds of clusters, tens of thousands of Redis-server nodes, and thousands of host machines, achieving high memory utilization through fine‑grained operations.
As business growth drives host memory usage higher, automated daily rebalancing is required to keep each host’s memory below a safe threshold and enable fast vertical scaling.
Manual rebalancing consumed significant DBA time and introduced stability risks, prompting the platform to implement an unattended resource‑balancing scheduler that runs at 5 AM.
Node selection follows a priority rule set: favor instances with many nodes, non‑P0 priority, slave nodes, medium specifications (1‑4 GB), and avoid picking multiple nodes from the same shard.
The migration workflow adds a slave node, verifies synchronization via info replication , promotes the slave with slaveof no one , confirms the switchover, then removes the original node and sends a completion notice.
Reliability checks ensure master‑slave sync is normal by confirming master_link_status:up , non‑zero master_repl_offset , and state=online with offset != 0 from both master and slave perspectives, with a re‑check after one minute.
The automated system maintains host memory at optimal levels, distributes cluster nodes across machines to reduce failure impact, and can also be used for node migration during host decommissioning, paving the way for further automation such as auto‑deployment and auto‑recovery.
DeWu Technology
A platform for sharing and discussing tech knowledge, guiding you toward the cloud of technology.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.