How to Achieve Seamless Horizontal Scaling with 2N Expansion and Keepalived

1. Expansion Scheme Analysis

1.1 Scaling Problem

Initially three databases A, B, C satisfy the business load, and data is evenly distributed using uid % 3 in the service layer. As user volume grows, the databases reach their optimal storage capacity; further inserts slow down CRUD operations and affect service response time.

Adding a new node D expands the cluster from three to four databases, but the sharding rule changes to uid % 4, causing most existing data to miss the original shards and requiring massive data migration.

For example, uid = 3 was stored in database A with 3 % 3 = 0 . After adding D, 3 % 4 = 3 stores it in D.

Approximately 90% of data must be migrated, creating heavy data pressure and service instability.

1.2 Shutdown Plan

Publish an announcement informing users of the upgrade schedule.

Stop the Service.

Perform offline data migration: split and re‑allocate data according to the new sharding algorithm.

Validate data consistency between old and new databases.

Update the service configuration to use uid % 4 .

Restart the Service.

Prepare rollback plans for each step.

The shutdown approach guarantees migration correctness but incurs service downtime, which hurts user experience.

1.3 Stop‑Write Plan

Enable read‑write separation; set databases to read‑only during migration.

Publish an upgrade notice.

Intercept write requests at the Service layer and return a uniform “service upgrading, read‑only” message.

Copy data from old to new databases.

Validate data consistency.

Update the sharding rule to uid % 4 via the configuration center (no service restart required).

Restore write capability.

Delete redundant data.

Prepare rollback plans for each step.

The stop‑write method reduces downtime but requires a long period of data copying and cleanup.

1.4 Log‑Based Scheme

The core idea is to use database logs for incremental migration.

Before migration, the application continues to read from the old database.

During upgrade, record every data modification (table, primary key, operation type) in a log.

Develop a tool to replay the log and apply changes to the new database.

After data copy, run a data‑validation tool to compare old and new databases.

Switch traffic to the new database only when validation succeeds.

If consistency is not 100%, keep the old database read‑only and continue replaying logs until full synchronization.

1.5 Dual‑Write Scheme (Small‑to‑Medium Data)

Implement dual writes using Canal or MQ:

Add the new database node.

Perform an initial data dump and import.

Enable Canal to capture changes from the old master and forward them to the new node.

Switch traffic via Nginx after synchronization.

Handle any missed writes during the switch with custom scripts.

Validate data completeness.

1.6 Smooth 2N Scheme (Large Data)

For large datasets, use a 2N architecture where each primary has a standby replica.

When scaling, promote the standby nodes to primary, adjust sharding rules, and clean up redundant data without stopping service.

2. Smooth 2N Expansion Practice

2.1 Implement Service‑Level Dynamic Scaling

Deploy MariaDB on the servers.

yum -y install wget
## Backup CentOS‑Base.repo
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -P /etc/yum.repos.d/ http://mirrors.aliyun.com/repo/epel-7.repo

yum clean all

yum makecache

Configure the MariaDB YUM source, install MariaDB, and run mysql_secure_installation to harden the instance.

Set up a replication user:

grant replication slave, replication client on *.* to 'replica'@'%' identified by 'replica';
flush privileges;

Configure master‑slave settings in /etc/my.cnf for both Server1 and Server2, ensuring different server-id and auto‑increment‑offset values.

Start replication with change master to ... and start slave, then verify show slave status\G reports Slave_IO_Running: Yes and Slave_SQL_Running: Yes.

2.1.2 MariaDB Double‑Master Sync

On Server1, add:

[mysqld]
server-id=1
log-bin=mysql-bin
relay-log=mysql-relay-bin
replicate-wild-ignore-table=mysql.%
replicate-wild-ignore-table=information_schema.%
log-slave-updates=on
slave-skip-errors=all
auto-increment-offset=1
auto-increment-increment=2
binlog_format=mixed
expire_logs_days=10

On Server2, set server-id=2 and auto-increment-offset=2. After restarting MariaDB, create the replica user on both servers, then run change master to ... and start slave on each side.

2.1.3 Install Keepalived for High Availability

Install Keepalived on both nodes: yum -y install keepalived Disable the firewall, set hostnames, and create /etc/keepalived/keepalived.conf with a virtual IP (e.g., 192.168.116.150) and real‑server entries pointing to the MariaDB instances. Example configuration (attributes removed):

global_defs {
    router_id vip1
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 111
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 6666
    }
    virtual_ipaddress {
        192.168.116.150
    }
}
virtual_server 192.168.116.150 3306 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 0
    protocol TCP
    real_server 192.168.116.140 3306 {
        notify_down "/usr/local/shell/mariadb.sh"
        weight 1
        TCP_CHECK {
            connect_timeout 10
            retry 3
            delay_before_retry 3
            connect_port 3306
        }
    }
}

Create /usr/local/shell/mariadb.sh containing pkill keepalived and make it executable.

2.1.4 Build Application Service

Introduce ShardingJDBC (a lightweight Java framework) to provide sharding at the JDBC layer. Configure data sources, sharding rules, and dynamic data‑source addition in code. Example dynamic addition snippet:

// Dynamic data source addition for scaling
Properties props = loadPropertiesFile("datasource1.properties");
DruidDataSource ds = (DruidDataSource) DruidDataSourceFactory.createDataSource(props);
ds.setRemoveAbandoned(true);
ds.setRemoveAbandonedTimeout(600);
ds.setLogAbandoned(true);
ds.setTimeBetweenConnectErrorMillis(60000);
ds.init();
OrchestrationShardingDataSource ods = SpringContextUtil.getBean("tradeSystemDataSource", OrchestrationShardingDataSource.class);
Map<String, DataSource> dsMap = ods.getDataSource().getDataSourceMap();
dsMap.put(DatasourceEnum.DATASOURCE_2.getValue(), ds);
// Refresh sharding rule configuration
SHARDING_RULE_DATASOURCE = newRule;
ods.renew(new DataSourceChangedEvent("/", dsMap));

Note limitations of ShardingJDBC (no sub‑queries, HAVING, UNION, multi‑row INSERT, etc.).

2.2 Realize Second‑Level 2N Expansion

Deploy additional virtual machines Server3 and Server4, install MariaDB, and configure new double‑master pairs (Server1‑Server3 and Server2‑Server4) following the same my.cnf settings with unique server-id values.

Set up Keepalived on the new nodes with distinct virtual IPs (e.g., 192.168.116.151) and corresponding real_server entries.

2.3 Clean Up and Verify

After migration, clean redundant data using SQL statements that retain only rows matching the new sharding rule (e.g., keep even accountNo on one node, odd on the other).

Run data‑validation tools to ensure consistency before switching traffic to the new cluster.

3. Keepalived High‑Availability Configuration Collection

Provide full keepalived.conf examples for Server1, Server2, Server3, and Server4, demonstrating virtual router IDs, priorities, virtual IPs, and real‑server definitions for both primary clusters.

All configuration files are stripped of style attributes, keeping only the essential directives.