Designing a Simple Master‑Slave HA System with Database‑Based Leader Election

1. Implemented Features

One master with multiple backups, automatic master election

Startup records are queryable

2. Prerequisites

A single database (e.g., MySQL, Redis, MongoDB) is required to store records; the design is database‑agnostic.

3. Design and Implementation

A table is created in the database with the following structure:

Column Name | Description
-----------|------------------------------
id         | Auto‑increment integer (easy to implement in NoSQL as well)
instance   | Stores the program instance name, e.g., app1, app2
startTime  | Instance startup timestamp
heartBeat  | Timestamp of the last heartbeat

Consider a cluster of two instances with a 1‑second heartbeat interval.

a. Instance Startup

When an instance starts, it inserts a record, for example: 0 | app1 | 2016-08-01 13:00:00 | 2016-08-01 13:00:00 When all instances have started, the table may look like:

0 | app1 | 2016-08-01 13:00:00 | 2016-08-01 13:01:00
1 | app2 | 2016-08-01 13:01:00 | 2016-08-01 13:01:00

b. Master Election

Rule: an instance considers itself the master only if its id is the smallest among rows with a valid heartbeat. In the example above, app1 is the master.

SQL used for election:

SELECT MIN(id) FROM table WHERE heartbeat > NOW() - heartbeat_interval;

c. Heartbeat

Each heartbeat cycle, every instance updates its own heartbeat field:

UPDATE table SET heartbeat = NOW() WHERE id = myid AND heartbeat > NOW() - heartbeat_interval;

If the update affects zero rows, the heartbeat has timed out, indicating a failure. A new record is then inserted, for example when app1 times out:

0 | app1 | 2016-08-01 13:00:00 | 2016-08-01 13:58:00
1 | app2 | 2016-08-01 13:01:00 | 2016-08-01 14:00:00
2 | app1 | 2016-08-01 14:00:00 | 2016-08-01 14:00:00

At the end of each heartbeat cycle, each instance checks whether it is still the master according to step b and performs a switch if necessary.