Using Orchestrator for Automatic MySQL Cluster Failover: Configuration and Test Cases

Parameter Description:

Reference: https://github.com/openark/orchestrator/blob/master/go/config/config.go

Purpose:

Use Orchestrator to configure automatic failover for a MySQL cluster.

Managed database instances (1 master 1 slave architecture):

10.186.65.5:3307
10.186.65.11:3307

Orchestrator related parameters:

"RecoveryIgnoreHostnameFilters": [],
"RecoverMasterClusterFilters": ["*"],
"RecoverIntermediateMasterClusterFilters": ["*"],
"ReplicationLagQuery": "show slave status",
"ApplyMySQLPromotionAfterMasterFailover": true,
"FailMasterPromotionOnLagMinutes": 1,

Test scenarios are executed on the raft-leader node.

Case 1:

Scenario:

Shut down the master and verify failover when the replication lag is less than FailMasterPromotionOnLagMinutes.

Operation:

# Confirm existing clusters
  orchestrator-client -c clusters
# View topology, cluster is 10.186.65.11:3307
  orchestrator-client -c topology -i 10.186.65.11:3307
# Stop master node
  ssh [email protected] "service mysqld_3307 stop"
# Confirm clusters again; original cluster splits into two
  orchestrator-client -c clusters
# View topology, now cluster is 10.186.65.5:3307
  orchestrator-client -c topology -i 10.186.65.5:3307

Conclusion:

Failover succeeded.

The new master has read_only and super_read_only disabled, allowing read‑write operations.

Case 2:

Scenario:

Shut down the master and verify failover when the replication lag exceeds FailMasterPromotionOnLagMinutes (configured as 1 minute).

Operation:

# Show current FailMasterPromotionOnLagMinutes value
  orchestrator -c dump-config --ignore-raft-setup | jq .FailMasterPromotionOnLagMinutes
# Confirm existing clusters
  orchestrator-client -c clusters
# View topology, cluster is 10.186.65.11:3307
  orchestrator-client -c topology -i 10.186.65.11:3307
# Create a delayed slave (e.g., 120 s)
  stop slave ;
  change master to master_delay=120;
  start slave ;
# or
  orchestrator-client -c delay-replication -i 10.186.65.5:3307 -S 120
# Wait 120 s
  sleep 120
# View topology, cluster remains 10.186.65.11:3307
  orchestrator-client -c topology -i 10.186.65.11:3307
# Stop master node
  ssh [email protected] "service mysqld_3307 stop"
# Confirm clusters and topology again
  orchestrator-client -c clusters
  orchestrator-client -c topology -i 10.186.65.11:3307

Conclusion:

No failover occurred.

When the slave lag exceeds FailMasterPromotionOnLagMinutes, failover is prevented.

Case 3:

Scenario:

Disable global recovery and shut down the master while lag is less than FailMasterPromotionOnLagMinutes.

Operation:

# Disable global recoveries
  orchestrator-client -c disable-global-recoveries
  orchestrator-client -c check-global-recoveries
# Confirm clusters and topology
  orchestrator-client -c clusters
  orchestrator-client -c topology -i 10.186.65.11:3307
# Stop master node
  ssh [email protected] "service mysqld_3307 stop"
# Confirm clusters and topology again
  orchestrator-client -c clusters
  orchestrator-client -c topology -i 10.186.65.11:3307

Conclusion:

No failover occurred.

Disabling global recoveries prevents automatic failover.

Summary:

After configuring Orchestrator, automatic failover can be controlled via parameters such as RecoveryIgnoreHostnameFilters, RecoverMasterClusterFilters, RecoverIntermediateMasterClusterFilters, as well as conditions like FailMasterPromotionOnLagMinutes and ReplicationLagQuery. When lag exceeds the configured minutes or global recovery is disabled, failover does not occur.

Testing scenarios are limited; further tests may be needed for specific cases.