Designing High‑Availability Systems: Multi‑Active Architectures, Failover, Monitoring, and SLO/SLI

In this article we explore the core techniques for designing highly available systems, covering multi‑active architectures, failover mechanisms, monitoring and alerting, and SLO/SLI error‑budget management.

1. Multi‑Active Architecture

1.1 Overview

The evolution of multi‑active architectures is illustrated as a progression:

┌─────────────────────────────────────────────────────────────────┐
│                     Single‑datacenter                         │
│  └─ Advantages: simple, low cost                               │
│  └─ Disadvantages: service unavailable on datacenter failure   │
│                     ▼                                         │
│                 Same‑city active‑active                       │
│  └─ Advantages: failover possible, low latency (<5ms)         │
│  └─ Disadvantages: cannot survive city‑level disasters       │
│                     ▼                                         │
│               Two‑city three‑center                         │
│  └─ Advantages: can handle city‑level disasters               │
│  └─ Disadvantages: higher cross‑city latency (20‑50ms), cost   │
│                     ▼                                         │
│                 Global multi‑active                         │
│  └─ Advantages: near‑by access, global coverage              │
│  └─ Disadvantages: strong consistency challenges, high cost   │
└─────────────────────────────────────────────────────────────────┘

1.2 Same‑City Active‑Active

Two datacenters (A and B) host identical services and share data via synchronous replication.

┌─────────────────────────────────────────────────────────────────┐
│                DNS/VPN                                         │
│                     │                                           │
│          ┌──────────┴──────────┐                            │
│          │                     │                            │
│          ▼                     ▼                            │
│   ┌────────────┐       ┌────────────┐                         │
│   │  Datacenter A│       │  Datacenter B│                         │
│   │ (primary)   │       │ (backup)    │                         │
│   │ + Application│◄────►│ + Application│                         │
│   │ + Redis      │  sync │ + Redis      │                         │
│   │ + MySQL      │       │ + MySQL      │                         │
│   └─────┬───────┘       └─────┬───────┘                         │
│         │                     │                               │
│         └───────┬─────────────┘                               │
│                 ▼                                           │
│          Data Synchronization (DRC/CDM)                        │
└─────────────────────────────────────────────────────────────────┘

Spring configuration for data synchronization:

@Configuration
public class DataSyncConfig {
    @Bean
    public DataSourceRouting dataSourceRouting() {
        return new DataSourceRouting();
    }
    // MySQL semi‑sync replication
    //INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';
    //SET GLOBAL rpl_semi_sync_slave_enabled = 1;
    // Redis sync configuration
    @Bean
    public RedisSyncService redisSyncService() {
        return new RedisSyncService(
            sourceRedis: "redis://机房A:6379",
            targetRedis: "redis://机房B:6379"
        );
    }
}

@Service
public class Cross机房Service {
    public RpcResult callRemoteService(String serviceName, Object param) {
        // Prefer local datacenter
        String localEndpoint = getLocalEndpoint(serviceName);
        try {
            return rpcClient.call(localEndpoint, param);
        } catch (Exception e) {
            // Fallback to remote datacenter
            String remoteEndpoint = getRemoteEndpoint(serviceName);
            return rpcClient.call(remoteEndpoint, param);
        }
    }
}

1.3 Two‑City Three‑Center

┌─────────────────────────────────────────────────────────────────┐
│                     Main City                                   │
│  ┌───────────────────────────────────────────────────────┐   │
│  │                     Primary Data Center                │
│  │                     (Primary)                         │
│  │               ◄────►   Same‑city DR Site               │
│  └───────────────────────────────────────────────────────┘   │
│                     │                                           │
│               Asynchronous Replication                        │
│                     │                                           │
│                     ▼                                           │
│  ┌───────────────────────────────────────────────────────┐   │
│  │                     Remote DR Site                        │
│  └───────────────────────────────────────────────────────┘   │
│                     Backup City                               │
└─────────────────────────────────────────────────────────────────┘

1.4 Global Multi‑Active

┌─────────────────────────────────────────────────────────────────┐
│   China North   │   China East   │   North America   │   Europe   │
│   Write Unit1  │   Read Unit2  │   Read Unit3      │   Read Unit4│
│   ──────┬─────│   ──────┬─────│   ──────┬─────│   ──────┬─────│
│         │           │           │           │           │
│         └─────────────┴─────────────┴─────────────┘
│                     ▼
│               Data Synchronization (eventual consistency)
└─────────────────────────────────────────────────────────────────┘

Global data router implementation:

@Service
public class GlobalDataRouter {
    public DataUnit getDataUnit(String userId) {
        int unitId = Math.abs(userId.hashCode()) % DATA_UNIT_COUNT;
        return DataUnit.values()[unitId];
    }
    @Async
    public void syncToOtherUnits(String userId, Object data) {
        for (DataUnit unit : DataUnit.values()) {
            if (unit != currentUnit) {
                kafkaTemplate.send("data-sync-" + unit.name(), userId, data);
            }
        }
    }
}

2. Failover

2.1 Health‑Check Mechanism

@Component
public class HealthChecker {
    @Autowired
    private DataSource dataSource;
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    public HealthStatus check() {
        HealthStatus status = new HealthStatus();
        status.setStatus("UP");
        // Database check
        try {
            jdbcTemplate.execute("SELECT 1");
            status.setDatabase("UP");
        } catch (Exception e) {
            status.setDatabase("DOWN");
            status.setStatus("DOWN");
        }
        // Redis check
        try {
            redisTemplate.opsForValue().get("health-check");
            status.setRedis("UP");
        } catch (Exception e) {
            status.setRedis("DOWN");
        }
        // Disk check
        File disk = new File("/");
        long free = disk.getFreeSpace();
        if (free < 1024L * 1024L * 1024L) { // less than 1 GB
            status.setDisk("LOW");
        } else {
            status.setDisk("OK");
        }
        // CPU load check
        OperatingSystemMXBean os = ManagementFactory.getOperatingSystemMXBean();
        double load = os.getSystemLoadAverage();
        status.setCpuLoad(load);
        return status;
    }
}

@Configuration
public class ActuatorConfig {
    @Bean
    public HealthIndicator customHealth() {
        return () -> {
            if (isHealthy()) {
                return Health.up().build();
            }
            return Health.down()
                .withDetail("error", "Service unavailable")
                .build();
        };
    }
}

2.2 Automatic Failover Workflow

┌─────────────────────────────────────────────────────────────────┐
│               Detected Failure                                 │
│                     │                                           │
│                     ▼                                           │
│          Confirm Failure (3 consecutive health‑check failures) │
│                     │                                           │
│                     ▼                                           │
│          Mark as Unavailable (remove from service list)       │
│                     │                                           │
│                     ▼                                           │
│          Traffic Switch (route to healthy nodes)               │
│                     │                                           │
│                     ▼                                           │
│          Send Alert Notification                               │
│                     │                                           │
│                     ▼                                           │
│          Recovery Detection (periodic check of original node)   │
│                     │                                           │
│                     ▼                                           │
│          ┌───────────────┐   Yes   ┌───────────────────────┐   │
│          │  Restore OK?  │◄───────│   Continue Monitoring   │   │
│          └───────────────┘          └───────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Service registry implementation with weighted round‑robin selection and failure counting:

@Service
public class ServiceRegistry {
    private final Map<String, List<Server>> serviceMap = new ConcurrentHashMap<>();
    private final Map<String, AtomicInteger> failureCounts = new ConcurrentHashMap<>();

    public void register(String serviceName, Server server) {
        serviceMap.computeIfAbsent(serviceName, k -> new CopyOnWriteArrayList<>())
                  .add(server);
        failureCounts.putIfAbsent(server.getId(), new AtomicInteger(0));
    }

    public Server selectServer(String serviceName) {
        List<Server> servers = serviceMap.get(serviceName);
        if (servers == null || servers.isEmpty()) {
            throw new ServiceNotFoundException(serviceName);
        }
        List<Server> healthyServers = servers.stream()
            .filter(Server::isHealthy)
            .collect(Collectors.toList());
        if (healthyServers.isEmpty()) {
            throw new NoHealthyServerException(serviceName);
        }
        return weightedRoundRobin(healthyServers);
    }

    public void reportFailure(String serverId) {
        AtomicInteger count = failureCounts.get(serverId);
        if (count != null) {
            int failures = count.incrementAndGet();
            if (failures >= FAILURE_THRESHOLD) {
                markServerUnhealthy(serverId);
            }
        }
    }

    public void reportSuccess(String serverId) {
        AtomicInteger count = failureCounts.get(serverId);
        if (count != null) {
            count.set(0);
        }
    }

    private void markServerUnhealthy(String serverId) {
        for (List<Server> servers : serviceMap.values()) {
            for (Server server : servers) {
                if (server.getId().equals(serverId)) {
                    server.setHealthy(false);
                    alertService.sendAlert("Server " + serverId + " marked unhealthy");
                }
            }
        }
    }
}

2.3 Database Failover

@Configuration
public class DataSourceFailoverConfig {
    @Bean
    @Primary
    public DataSource dataSource() {
        HikariConfig config = new HikariConfig();
        config.setJdbcUrl("jdbc:mysql://master:3306/db");
        config.setUsername("user");
        config.setPassword("pass");
        config.setPoolName("master-pool");
        // Test query for failure detection
        config.setConnectionTestQuery("SELECT 1");
        config.setValidationTimeout(5000);
        return new HikariDataSource(config);
    }

    @Bean
    public DataSource slaveDataSource() {
        HikariConfig config = new HikariConfig();
        config.setJdbcUrl("jdbc:mysql://slave:3306/db");
        config.setUsername("user");
        config.setPassword("pass");
        config.setPoolName("slave-pool");
        return new HikariDataSource(config);
    }
}

@Component
public class ReplicationFailoverController {
    public void initiateFailover() {
        // 1. Stop writes to master
        setMasterReadOnly();
        // 2. Wait for slave to catch up
        waitForReplicationComplete();
        // 3. Promote slave to master
        promoteSlaveToMaster();
        // 4. Update routing configuration
        updateRoutingConfiguration();
        // 5. Notify applications
        notifyApplicationOfFailover();
    }
}

3. Monitoring & Alerts

3.1 Monitoring Pillars

┌─────────────────────────────────────────────────────────────────┐
│                     Metrics (system, app, business)          │
│   • System: CPU, memory, disk, network                         │
│   • Application: QPS, latency, error rate                     │
│   • Business: order count, transaction amount, DAU            │
│                     Logging (app, access, security)          │
│                     Tracing (distributed, call‑chain, bottlenecks)│
│                     Events (deployment, scaling, alerts)       │
└─────────────────────────────────────────────────────────────────┘

3.2 Prometheus + Grafana

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'spring-boot-app'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['app:8080']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '(.*):\d+'
        replacement: '${1}'

  - job_name: 'redis'
    static_configs:
      - targets: ['redis:6379']

  - job_name: 'mysql'
    static_configs:
      - targets: ['mysql:3306']

# alerts.yml
groups:
  - name: spring-boot-alerts
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) > 0.1
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value }} errors/sec"

      - alert: HighLatency
        expr: histogram_quantile(0.95, sum(rate(http_server_requests_seconds_bucket[5m])) by (le)) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High latency detected"
          description: "P95 latency is {{ $value }}s"

      - alert: HighMemoryUsage
        expr: jvm_memory_used_bytes / jvm_memory_max_bytes > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage"
          description: "Memory usage is {{ $value | humanizePercentage }}"

Custom business metrics recorded with Micrometer:

@Service
public class BusinessMetricsService {
    private final MeterRegistry meterRegistry;
    public BusinessMetricsService(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
    }
    public void recordOrder(Order order) {
        Counter.builder("orders.total")
            .tag("type", order.getType())
            .tag("status", order.getStatus())
            .register(meterRegistry)
            .increment();
        DistributionSummary.builder("orders.amount")
            .tag("currency", order.getCurrency())
            .register(meterRegistry)
            .record(order.getAmount().doubleValue());
    }
    public void recordUserLogin(Long userId, String source) {
        Counter.builder("user.login")
            .tag("source", source)
            .register(meterRegistry)
            .increment();
    }
}

3.3 Tracing

@Configuration
public class TracingConfig {
    @Bean
    public OpenTelemetry openTelemetry() {
        return OpenTelemetrySdk.builder()
            .setTracerProvider(TracerProvider.builder()
                .addSpanProcessor(SimpleSpanProcessor.builder(
                    OtlpGrpcSpanExporter.builder()
                        .setEndpoint("http://jaeger:4317")
                        .build())
                    .build())
                .build())
            .build();
    }
    @Bean
    public Tracer tracer(OpenTelemetry openTelemetry) {
        return openTelemetry.getTracer("order-service");
    }
}

@Service
public class OrderService {
    private final Tracer tracer;
    public Order createOrder(OrderRequest request) {
        Span span = tracer.spanBuilder("createOrder")
            .setParent(Context.current())
            .startSpan();
        try (Scope scope = span.makeCurrent()) {
            span.setAttribute("order.amount", request.getAmount().doubleValue());
            span.setAttribute("user.id", request.getUserId());
            Order order = doCreateOrder(request);
            span.setAttribute("order.id", order.getId());
            span.setStatus(StatusCode.OK);
            return order;
        } catch (Exception e) {
            span.setStatus(StatusCode.ERROR, e.getMessage());
            span.recordException(e);
            throw e;
        } finally {
            span.end();
        }
    }
}

// Spring Cloud Sleuth auto‑instrumentation requires only the dependency:
// implementation "org.springframework.cloud:spring-cloud-starter-sleuth"

4. SLO & Error Budget

4.1 SLO/SLI/SLA Definitions

┌─────────────────────────────────────────────────────────────────┐
│ SLI (Service Level Indicator) – quantitative metric of service │
│   • Examples: availability, latency, throughput, error rate   │
│   • Formula: SLI = good_requests / total_requests            │
│                                                               │
│ SLO (Service Level Objective) – target value for an SLI      │
│   • Examples: 99.9% monthly availability, P99 latency <500ms│
│   • Set based on business needs and cost trade‑offs          │
│                                                               │
│ SLA (Service Level Agreement) – legal contract with client    │
│   • Example: "Service availability not less than 99.5%"      │
│   • Usually looser than SLO and may include compensation    │
└─────────────────────────────────────────────────────────────────┘

SLO service implementation:

@Service
public class SLOService {
    private static final double AVAILABILITY_SLO = 99.9; // 99.9%
    private static final double LATENCY_SLO = 99.0;      // P99
    private static final Duration LATENCY_THRESHOLD = Duration.ofMillis(500);

    private final MeterRegistry meterRegistry;
    private final AtomicLong totalRequests = new AtomicLong(0);
    private final AtomicLong successfulRequests = new AtomicLong(0);
    private final AtomicLong slowRequests = new AtomicLong(0);

    public void recordRequest(boolean success, Duration latency) {
        totalRequests.incrementAndGet();
        if (success) {
            successfulRequests.incrementAndGet();
        }
        if (latency.compareTo(LATENCY_THRESHOLD) > 0) {
            slowRequests.incrementAndGet();
        }
    }

    public SLOStatus getSLOStatus() {
        SLOStatus status = new SLOStatus();
        double availability = (successfulRequests.get() * 100.0) / totalRequests.get();
        status.setCurrentAvailability(availability);
        status.setTargetAvailability(AVAILABILITY_SLO);
        double latencyP99 = (slowRequests.get() * 100.0) / totalRequests.get();
        status.setCurrentLatencyP99(latencyP99);
        status.setTargetLatencyP99(LATENCY_SLO);
        long totalErrorBudget = (long) (totalRequests.get() * (100 - AVAILABILITY_SLO) / 100);
        long usedErrorBudget = totalRequests.get() - successfulRequests.get();
        long remainingErrorBudget = totalErrorBudget - usedErrorBudget;
        status.setTotalErrorBudget(totalErrorBudget);
        status.setUsedErrorBudget(usedErrorBudget);
        status.setRemainingErrorBudget(remainingErrorBudget);
        return status;
    }
}

4.2 Error‑Budget Management Flow

┌─────────────────────────────────────────────────────────────────┐
│                     Month Start                                 │
│   Calculate monthly error budget (e.g., 10 000 errors for 10 M│
│   requests with 99.9% SLO)                                      │
│                     ▼                                           │
│   Daily Monitoring ──► Compare consumption rate with plan      │
│       ├─ Normal consumption → continue as usual                  │
│       ├─ Rapid consumption → raise attention, limit changes      │
│       └─ Exhausted → freeze changes, perform urgent fixes       │
└─────────────────────────────────────────────────────────────────┘

4.3 Toil Automation

@Service
public class ToilAutomationService {
    // 1. Automatic alert handling
    @KafkaListener(topics = "alerts")
    public void handleAlert(Alert alert) {
        if (isCommonAlert(alert)) {
            autoRemediate(alert);
        } else {
            notifyOnCallEngineer(alert);
        }
    }

    // 2. Automatic scaling
    @Scheduled(fixedRate = 60000)
    public void checkScaling() {
        double cpuUsage = getCurrentCpuUsage();
        double targetInstances = Math.ceil(cpuUsage / TARGET_CPU_UTILIZATION);
        if (targetInstances > currentInstances) {
            scaleUp((int) targetInstances - currentInstances);
        } else if (targetInstances < currentInstances - MIN_FREE_INSTANCES) {
            scaleDown(currentInstances - (int) targetInstances);
        }
    }

    // 3. Automated disaster‑recovery failover
    public void autoFailover() {
        if (!shouldFailover()) return;
        if (!confirmFailure()) return;
        executeFailover();
    }
}