This article dissects why static DNS-based traffic scheduling fails at ten-million-QPS scale and how layered dynamic scheduling — across access, gateway, service, and data layers — combined with unitized routing, automated decision loops, and rigorous drill practices enables precise, reversible, and safe failover.
This article provides a comprehensive practical guide to MySQL master-slave replication, covering synchronization principles, three replication modes, delay causes and troubleshooting, read-write splitting implementation with common pitfalls, failover strategies, and five production pitfalls to avoid.
The article breaks down Redis‑Cluster failover into detection, election, failover, and client perception stages, explains the timing bottlenecks of each, and provides concrete server‑side and Lettuce client configurations that shrink end‑to‑end recovery to under ten seconds.
This article dissects Redis Sentinel’s high‑availability workflow—from failure detection, SDOWN/ODOWN states, and quorum logic to leader election, replica promotion, and configuration propagation—while illustrating each step with a real‑world e‑commerce cache case, detailed configuration snippets, Kubernetes deployment patterns, Spring Boot integration, and operational playbooks for observability and fault‑injection testing.
This article provides a comprehensive, hands‑on guide to MySQL master‑slave replication, covering the underlying architecture, binlog formats, GTID and semi‑synchronous modes, detailed configuration steps, thread workflows, common failure scenarios with step‑by‑step diagnostics, and practical monitoring and failover scripts.
The article explains that high availability means designing systems to keep serving after failures through redundancy, fault detection, failover, degradation, and clear RTO/RPO goals, rather than trying to avoid any downtime altogether.
This article explains why a single‑node Redis deployment is a single‑point‑of‑failure and walks through building a highly available Redis cluster using master‑slave replication, Sentinel monitoring and automatic failover, split‑brain prevention, production deployment guidelines, common pitfalls, and client‑side connection strategies.
The article examines why cross‑datacenter replication must evolve from simple asynchronous copying to semi‑synchronous and layered strategies at the ten‑million‑QPS scale, detailing latency, RPO, bandwidth costs, failover complexities, and practical selection guidelines for each business tier.
This article explains how to build highly available services by comparing single‑datacenter, same‑city active‑active, two‑city three‑center, and global multi‑active architectures, then details health‑check mechanisms, automatic failover workflows, Prometheus‑Grafana monitoring, and SLO/SLI error‑budget management with concrete code examples.
This guide walks through Redis Sentinel’s architecture, explains subjective and objective down states, details the leader election and failover workflow, shows step‑by‑step configuration of a three‑node Sentinel cluster, client integration in Python and Java, and provides best‑practice recommendations, monitoring metrics, and troubleshooting tips.
This article walks through building a simple, cost‑effective high‑availability solution for Nginx using Keepalived’s VRRP‑based VIP failover, covering environment setup, configuration of master and backup nodes, health‑check scripts, testing procedures, troubleshooting tips, and rollback steps.
The article explains how to build an enterprise‑grade payment gateway that unifies over 50 providers, performs millisecond‑level smart routing, handles failover, dynamic fee calculation, automated merchant onboarding, sharded storage, and comprehensive monitoring to sustain millions of transactions per day.
This article provides a comprehensive technical analysis of Redis Cluster, covering its data sharding via 16,384 hash slots, master‑slave replication, automated failover mechanisms, and the essential role of smart clients in routing, along with practical engineering guidelines and production‑grade code examples.
This guide explains how to achieve uninterrupted MySQL service by using semi‑sync replication, configuring MHA for automatic failover, deploying Orchestrator for topology management, and applying VIP‑based scripts and GTID settings to guarantee zero data loss during master switches.
This guide walks through the limitations of Redis Sentinel, introduces Redis Cluster’s sharding architecture, shows how to configure multiple nodes on a single server, demonstrates cluster creation, key slot calculation, basic CRUD tests, and explains automatic failover behavior when a master node goes down.
This step‑by‑step guide shows how to design, install, configure, start, and verify a Redis Sentinel high‑availability setup with one master, two replica nodes, and three sentinel monitors, including client connection examples and common troubleshooting tips.
Learn how to set up Redis Sentinel for high‑availability caching, covering prerequisites, anti‑patterns, detailed configuration of master, replicas and Sentinel nodes, firewall rules, monitoring, failover testing, troubleshooting, performance tuning, backup, rollback and best practices—all achievable within a 30‑minute deployment.
This comprehensive guide explains how to diagnose MySQL master‑slave replication lag, enable GTID mode, configure semi‑synchronous replication, optimize multi‑threaded replication, set up monitoring and alerting with Prometheus, and perform zero‑data‑loss failover using tools like Orchestrator and custom scripts.
This article walks through a systematic emergency response for Redis outages, explains how Redis Cluster promotes a replica, quantifies the typical data‑loss window from hundreds of milliseconds to several seconds, and provides detailed persistence configurations (RDB, AOF, and hybrid) to minimise downtime and data loss.
This comprehensive guide explains PostgreSQL streaming replication concepts, required environment, primary and standby configuration commands, verification queries, failover procedures, and production best‑practice recommendations, enabling you to build a reliable high‑availability database cluster.
This guide walks you through installing and configuring DRBD and keepalived on two Linux nodes to achieve real‑time block‑level file synchronization, automatic primary/secondary role switching, and high‑availability failover for services such as PostgreSQL, including troubleshooting common issues like split‑brain and busy mounts.
This guide walks through the reasons for MySQL master‑slave replication, explains its core mechanisms, details step‑by‑step environment planning, configuration, data initialization, replication setup, monitoring, failover with MHA, read‑write splitting using ProxySQL, performance tuning, troubleshooting, and best‑practice recommendations for enterprise‑grade high availability.
This article provides a comprehensive guide to Redis Sentinel, covering its purpose, architecture, monitoring functions, discovery mechanisms, failover process, leader election, configuration options, and practical commands for achieving reliable high‑availability in Redis deployments.
This article explains how to achieve high availability in internet systems by designing redundant components and automatic failover mechanisms across layers such as load balancers, reverse proxies, microservices, middleware, databases, and messaging, illustrating concepts with diagrams of architectures, clustering, leader election, and practical tools like keepalived, Zookeeper, Redis Sentinel, and Kafka.
This article explains how Nginx's master‑slave (primary‑backup) setup, combined with Keepalived and a virtual IP, provides high‑availability for web and API services by automatically detecting failures, shifting the VIP, and allowing the backup server to take over without service interruption.
This article explores how to embed load‑balancer concepts such as intelligent request distribution, health checks, automatic failover, and dynamic strategy adjustment directly into PHP applications using algorithms like weighted round‑robin, response‑time balancing, and circuit‑breaker patterns, providing code examples and practical deployment scenarios.
Understanding high‑availability architectures involves mastering replication and fail‑over, balancing RTO and RPO, and choosing among four patterns—Active‑Standby, Active‑Active, Cold Standby, and Hot Standby—each with distinct synchronization, load‑balancing, and cost considerations for reliable system design.
Learn step-by-step how to install and configure DRBD for real-time block device replication, set up keepalived for virtual IP failover, and manage primary/secondary roles, including troubleshooting split-brain, to achieve seamless file synchronization and high availability between two Linux servers.
Redis Sentinel provides a robust high‑availability solution by monitoring master‑slave clusters, automatically detecting failures, electing leaders, and performing failover, while using quorum voting, Pub/Sub communication, and configuration provisioning to ensure seamless master promotion and client redirection without manual intervention.
This article explains Redis Sentinel’s role as the official high‑availability solution, detailing its monitoring, notification, automatic failover mechanisms, discovery processes, connection types, down‑state classifications, failover steps, leader election, master selection rules, and data consistency guarantees.
Bilibili’s comment service, a critical component for user interaction, faces massive read‑write traffic that can overwhelm TiDB, so the team built a multi‑level storage architecture using Redis sorted‑sets for indexes and a custom Taishan KV store, adding automatic degradation, consistency mechanisms, and hedging policies to ensure high availability and performance.
The article explains how elastic architecture, through auto‑scaling and failover mechanisms, dynamically adjusts resources and ensures continuous service during traffic spikes and component failures, improving cost efficiency, reliability, and operational stability for modern cloud‑based applications.
This article investigates a Redis Sentinel failover anomaly in version 7.4.0 where the sentinel repeatedly elects a failed master, explains the underlying s_down/o_down states, examines network, configuration, and ACL settings, and resolves the issue by adjusting Pub/Sub permissions to allow proper failover.
This article explores how Alibaba’s Nuwa service and the open‑source ZooKeeper implement distributed read‑write locks using sequential files, detailing the challenges of maintaining cversion order during failover and the community’s solutions to ensure consistency and reliability.
This guide outlines ten practical strategies—including rollback testing, safe handling of destructive commands, prompt customization, robust backup and verification, production environment discipline, thorough handover, proactive monitoring, cautious auto‑failover, meticulous execution, and simplicity—to help operations engineers dramatically reduce system outages and improve reliability.
This article explains Redis master‑slave replication, synchronization steps, handling of network partitions, and how Sentinel provides automatic failover through monitoring, leader election, and notification, offering strategies to reduce master load and ensure high availability.
This article presents a comprehensive design and implementation for achieving MySQL high availability by replacing the single‑master architecture with Orchestrator‑driven automatic failover, integrating DBProxy for transparent routing, and addressing topology changes and data compensation to ensure continuous, reliable service.
This article walks through real‑world Oracle RAC failures caused by misconfigured VIP failover and SCAN IP load‑balancing, explains how to diagnose the symptoms, provides correct TAF and listener settings, and highlights essential configuration tips to ensure reliable high‑availability operation.
This article explains Redis Sentinel’s role in providing monitoring, notifications, automatic failover, and configuration updates to achieve high availability, detailing its heartbeat mechanism, master‑down detection, leader election, failover selection criteria, and the trade‑offs of using this solution.
This guide explains how to configure Istio locality‑based load balancing and failover across Kubernetes zones and regions, providing hands‑on scripts, YAML examples, and validation steps to ensure resilient microservice traffic routing in cloud‑native environments.
Redis Sentinel provides automatic monitoring, fault detection, and failover for Redis master‑slave clusters, enabling high availability by electing a new master when the original fails, using sdown/odown states, quorum voting, and pub/sub communication to keep services running with minimal downtime.
This article examines Alibaba's internal Doris KV storage system, detailing why large companies build proprietary data products, the project's kickoff criteria, the two‑layer architecture, virtual‑node routing, failover mechanisms, and cluster scaling strategies for massive KV workloads.
The article investigates why Go applications unexpectedly send AAAA DNS queries to a secondary nameserver, tracing the issue to the built‑in resolver’s handling of non‑recursive responses from a NetScaler proxy, and recommends using the cgo resolver, enabling recursion, or forcing IPv4 to eliminate the added latency.
This guide walks through identifying master and replica pods, performing graceful switchover, triggering automated failover, and reinitializing replicas in a StackGres 1.6 cluster using kubectl and patronictl commands, with practical notes and safety warnings.
A recent Redis deployment faced memory expansion, a master‑slave switch, and unexpected data loss when the new master entered read‑only mode, prompting a deep dive into sentinel behavior, maxmemory settings, and replica‑ignore‑maxmemory nuances to prevent similar failures.
This article introduces Redis Sentinel as the official high‑availability solution for Redis, explains its core functions, provides configuration examples, compares three ways to receive failover notifications (script, client subscription, and indirect service), and offers design recommendations for robust production deployments.
This article provides a comprehensive guide to Redis Cluster, covering its sharding mechanism, hash slot mapping, replication and automatic failover, client data location, slot reassignment, MOVED/ASK redirection, communication overhead, and practical tuning tips for large‑scale deployments.
This guide walks through preparing two CentOS virtual machines, installing Nginx, explaining high availability concepts, and configuring Keepalived with VRRP to achieve automatic failover and virtual IP management for a resilient Nginx service.
This guide walks through preparing two CentOS VMs, installing Nginx, explaining high‑availability concepts, and configuring Keepalived with VRRP and a health‑check script to achieve automatic failover of Nginx services via a virtual IP.
An incident report details how a scheduled machine reboot on Alibaba Cloud triggered a Flink TaskManager failover, leading to excessive data replay, increased ES pressure, and significant business latency, and explains the root cause involving disabled checkpoints and timestamp‑based offset consumption.
This article explains how to design a high‑availability SMS system by identifying stability bottlenecks, defining reliability goals, implementing failover strategies for Redis, MySQL and external services, establishing a comprehensive observability framework, and measuring key quality metrics to ensure 99.99% uptime.
This article explains how Orchestrator selects a candidate replica during MySQL master failover, detailing the GetCandidateReplica and RegroupReplicasGTID functions, their sorting logic, promotion rules, GTID-based regrouping, and differences from MHA, while highlighting potential data loss issues and related bugs.
This article explains the complete DeadMaster recovery workflow of Orchestrator, detailing how the system selects the appropriate check‑and‑recover function, handles emergency grace periods, reads topology information, registers recovery attempts, validates promotion constraints, executes the actual failover, and runs post‑recovery hooks, with extensive Go code examples.
This article walks through a simulated MySQL 3307 cluster failure, examines Orchestrator's source code to explain the ContinuousDiscovery loop, discovery queues, health ticks, caretaking tasks, raft coordination, topology snapshots, and the logic distinguishing UnreachableMaster from DeadMaster states.
The article analyzes a MySQL MGR cluster failover incident caused by a VM pause and system time jump, explains the underlying detection mechanisms, presents detailed source code excerpts, and demonstrates through debugging that time changes alone do not trigger MGR failover.
This article demonstrates how to configure the open-source Orchestrator tool for automatic MySQL cluster failover, explains key parameters, and presents three test cases covering normal failover, lag‑induced prevention, and the effect of disabling global recoveries.
The article explains how Redis Sentinel provides high availability by monitoring master‑slave instances, detecting failures through periodic pings, distinguishing subjective and objective down states, performing quorum arbitration, and automatically promoting a slave to master to ensure continuous service.
This guide walks through upgrading a single‑node Redis deployment to a high‑availability setup by building a master‑slave cluster, configuring Sentinel services, testing replication and failover, and enabling client auto‑detection of master changes, all using Docker containers and configuration files.
This guide walks through installing a Redis cluster with three masters and three slaves using local ports, explains slot allocation, key hashing, gossip communication, failover, node addition, resharding, and best practices for high availability, while providing practical commands and configuration examples.
The Apache Pulsar 2.10.0 release introduces automatic cluster failover, lazy‑loading producers, new TableView support, enhanced broker interceptors, enriched client authentication, Etcd metadata storage, and numerous bug fixes, offering developers and operators a more flexible and performant messaging platform.
This article explains Redis Sentinel's architecture, initialization, inter‑process communication, node discovery, down‑detection, leader election, and the step‑by‑step failover process for automatically promoting a slave to master and reconfiguring the old master.
High availability (HA) is essential for distributed systems, requiring redundancy and automatic failover across each architectural layer—from client to proxy, gateway, business logic, cache, and storage—to minimize downtime, achieve desired “nines” of uptime, and prevent cascading failures such as service snowballing.
This guide shows how to use the Bitnami postgresql‑repmgr Docker image, which includes the Replication Manager tool, to create a high‑availability PostgreSQL cluster with streaming replication, failover, persistent storage, Docker networking, optional TLS, and upgrade procedures.
This article explains MySQL high availability concepts, how master‑slave replication works, how to measure and mitigate replication lag, and compares reliable‑first and availability‑first failover strategies with practical experiments on binlog formats.
This article explains Redis Sentinel’s role in achieving high availability, details its core functions, underlying Raft‑based algorithm, configuration parameters, practical setup steps, fault‑tolerance mechanisms, quorum and majority calculations, and demonstrates failover and recovery scenarios with real command‑line examples.
The MHA Re-Edition tool revives the discontinued MHA manager for MySQL, adding GTID‑based failover, password‑only SSH authentication, lightweight binaries, VIP migration, WeChat alerts, remote‑card reboot, and detailed configuration options, with step‑by‑step deployment instructions and sample app1.cnf parameters for high‑availability clusters.
The article explains why MHA’s default long‑connection ping cannot detect a hung MySQL instance, introduces three configurable ping_type options (select, connect, insert), shows how to modify the configuration and test the behavior with Perl code and shell scripts, and details the internal health‑check workflow.
This guide walks through setting up an Undermoon Redis cluster using Docker Compose, covering prerequisites, building images, registering proxies, creating and scaling the cluster, and demonstrating automatic failover with concrete command‑line examples.
This article explains ProxySQL's built‑in databases, key configuration tables such as mysql_servers, mysql_users, mysql_replication_hostgroups, mysql_group_replication_hostgroups, and mysql_query_rules, and demonstrates how to set up read/write splitting and automatic failover for MySQL primary‑replica and group replication environments.
This guide demonstrates how to deploy a ProxySQL instance alongside six MySQL servers (three for traditional replication and three for MySQL Group Replication), configure users, set up read/write splitting rules, and enable automatic failover for both replication topologies.
This tutorial explains how to achieve high availability with keepalived by installing the software, configuring VRRP virtual IPs, setting up master and backup nodes, starting the service, and verifying failover through VIP testing on Linux systems.
This article examines the limitations of Redis master‑slave replication, explains how Redis Sentinel addresses those issues with monitoring, notification, and automatic failover, and provides detailed configuration commands, discovery mechanisms, and step‑by‑step failover procedures for building a highly available Redis deployment.
The article explains how MHA handles a crashed MySQL master by preserving binary logs, selecting the most up‑to‑date slave, applying relay logs, promoting a slave to master, and provides detailed commands for creating replication users and configuring master‑slave replication.
This article explains how to use Keepalived together with PostgreSQL to build a two‑node high‑availability cluster, covering Keepalived's VRRP mechanism, host planning, installation steps, asynchronous master‑slave replication configuration, monitoring scripts, and detailed failover drills.
This article explains Redis Sentinel’s role in high‑availability deployments, covering its architecture, monitoring and notification mechanisms, automatic failover process, configuration steps for master‑slave and sentinel nodes, and practical guidelines for building a reliable Redis cluster.
This article explains how to configure Nomad constraints, particularly the datacenter attribute, to limit DTLE client failover within specific data centers, detailing job structure, attribute selection, deployment steps, and verification of task migration.
This article explains how to build a highly available Redis service using Redis Sentinel, discusses common failure scenarios, compares several architectural options from a single instance to a three‑node Sentinel setup, and provides practical tips such as using virtual IPs for seamless client access.
This article explains how to build a highly available Redis service using Redis Sentinel, covering failure scenarios, design principles, four architectural options, the final three‑Sentinel solution with virtual IP, and practical operational tips for ensuring continuous service.
This guide explains what Redis Sentinel is, its architecture, how to configure it, and demonstrates both slave and master failure recovery as well as configuring multiple Sentinel instances for high availability.
This guide explains what Redis Sentinel is, its architecture, how to configure it in a master‑replica environment, and demonstrates handling both slave and master failures with detailed log examples, plus tips for running multiple Sentinel instances for high availability.
This guide explains Keepalived’s role in Linux load‑balancing and high‑availability, covering its VRRP‑based architecture, core modules, layered operation, configuration syntax, practical deployment with Nginx, common split‑brain issues, and advanced settings such as nopreempt and multicast conflict resolution.
This article explains what Redis Sentinel is, its architecture, how to configure sentinel.conf, monitor master and slaves, handle slave and master failures, and set up multiple sentinel instances for robust high‑availability in a Redis master‑slave environment.
This guide explains how to achieve high availability for Nginx by installing and configuring Keepalived with VRRP, detailing preparation steps, configuration file examples, health‑check scripts, and failover testing on two Linux servers.
This article explains Redis Sentinel's role in providing high availability by detailing how Sentinels discover each other, detect master failures, elect a new master, and communicate state using the gossip protocol, offering a clear conceptual overview for database practitioners.
This guide explains what Redis Sentinel is, its architecture, how to configure it in a master‑slave environment, and demonstrates handling slave and master failures as well as setting up multiple Sentinel instances for robust high‑availability Redis clusters.
This guide explains how to achieve high availability for Nginx by deploying a dual‑machine keepalived setup, covering the concepts of HA, VRRP, configuration of keepalived on master and backup nodes, a health‑check script, and step‑by‑step commands to test automatic failover.
This guide explains how to achieve high availability for Nginx using Keepalived by setting up a double‑machine hot‑standby configuration, installing necessary packages, configuring VRRP‑based failover, writing health‑check scripts, and testing automatic VIP takeover when the primary server fails.
This article explains how Redis Sentinel provides automatic failover, monitoring, and client notification, outlines its configuration, compares three methods for receiving failover messages, and recommends a service‑level indirect reception design for scalable, low‑cost high availability.
This article explains what Redis Sentinel is, its architecture, how to configure it in a master‑slave environment, and demonstrates handling of slave and master failures as well as setting up multiple Sentinel instances for high availability.
This article examines the limitations of Redis master‑slave replication, such as manual failover and single‑node bottlenecks, and provides an in‑depth exploration of Redis Sentinel’s architecture, configuration parameters, detection mechanisms, automatic failover process, and best‑practice recommendations for achieving high availability.
A sysadmin narrates how he monitors four Redis nodes, detects master failure with PING, promotes a slave using SLAVEOF, reconfigures the remaining replicas, and ultimately automates the entire process with a custom Sentinel program and a multi‑node Sentinel cluster for high availability.
This article explains SQL Server's evolution from legacy high‑availability solutions to the modern AlwaysOn architecture, detailing its data‑synchronization process, synchronous and asynchronous commit modes, failover scenarios, and practical deployment recommendations for enterprises handling both moderate and terabyte‑scale workloads.
This article explains common database high‑availability designs—including master‑slave, master‑master, and automatic failover architectures—their topologies, advantages, disadvantages, and practical considerations such as replication lag, manual intervention, and data consistency challenges.
This article provides a comprehensive analysis of Redis Cluster, covering node and slot assignment, command execution, resharding, redirection, failover, gossip messaging, and communication overhead, while explaining why clustering is needed, how it works, and how to deploy and manage it effectively.
This article explains high availability fundamentals, defines availability, outlines design targets, presents common metrics such as MTBF, MTTR, MTTF, SA, RPO, RTO, discusses CAP theory, essential design elements, and answers practical Q&A on cost, architecture, fault tolerance, testing, and implementation guidance.
An operations engineer narrates how he monitors a four‑node Redis cluster, detects master failure with continuous PINGs, promotes a slave to master, reconfigures replicas, and automates the entire process with a sentinel program and a sentinel cluster for high availability.
This article explains Redis Sentinel's role in providing high availability for Redis by detailing its architecture, monitoring, leader election, automatic failover process, configuration of master‑slave instances and sentinel nodes, and how clients can subscribe to sentinel events.
This article provides a comprehensive analysis of Redis Cluster, covering node and slot assignment, command execution, resharding, redirection, fault‑tolerance, gossip communication, scaling strategies, configuration limits, and practical code examples for building and operating a high‑availability sharded Redis deployment.
System health checks, akin to medical exams, are vital for modern IT infrastructure, using active and passive monitoring, failover strategies, and tools like Spring Boot Actuator to detect hardware, network, load, or software issues, prevent single points of failure, and ensure continuous high‑availability service operation.
This article explains how to build a highly available Redis service by analyzing common failure scenarios, evaluating single‑instance, master‑slave with one or multiple Sentinel processes, and ultimately recommending a three‑Sentinel architecture combined with a virtual IP for seamless client usage.
