Multi-AZ Deployment and High Availability Practices for Tencent Cloud Elasticsearch

This article introduces the principles and practical steps for deploying Tencent Cloud Elasticsearch (ES) clusters across multiple Availability Zones (AZs) to achieve service stability and high availability.

Data Nodes : When a multi-AZ architecture is selected, the number of data nodes must be a multiple of the AZ count (e.g., 3, 6, 9 for three AZs). The console automatically distributes data nodes evenly across the AZs and tags each node with its zone attribute, enabling zone‑aware shard allocation.

Dedicated Master Nodes : For multi‑AZ clusters, three dedicated master nodes are required. In a three‑AZ deployment, one master node is placed in each AZ. In a two‑AZ deployment, an additional hidden AZ is used to host a master node to avoid a situation where both masters reside in the same AZ.

Index Replica Shards : To maintain data integrity when an AZ fails, each index must have at least one replica. For three‑AZ clusters, a minimum of two replicas is recommended so that the service remains functional even if two AZs become unavailable.

Implementation Mechanism :

Enable node attribute awareness by setting node.attr.zone_id:shanghai-3 (or the appropriate zone) in elasticsearch.yml or via the startup command ./bin/elasticsearch -Enode.attr.zone_id=shanghai-3 .

Configure the cluster to use the zone attribute for shard allocation: cluster.routing.allocation.awareness.attributes=zone_id .

Activate force awareness to prevent disk exhaustion in the remaining AZ when one AZ is down: cluster.routing.allocation.awareness.force.zone_id.values=shanghai-3,shanghai-4 .

With these settings, ES distributes primary and replica shards across different zones. If a zone fails, ES rebuilds the missing shards in the surviving zone(s), and force awareness ensures sufficient disk resources.

Upgrading a Single‑AZ Cluster to Multi‑AZ :

The article outlines two scenarios:

If the original single‑AZ cluster has no dedicated master nodes, the upgrade process involves adding ordinary nodes and dedicated masters in the new AZ(s), then converting the original data nodes to zone‑aware data nodes.

If the original cluster already has three dedicated masters, the upgrade adds data nodes and masters in the new AZ(s) and decommissions excess masters from the original zone.

Both scenarios risk temporary unavailability if the original AZ fails during the transition. To mitigate this, the improved process adds two dedicated masters in the new AZ before removing any masters from the original zone, ensuring that a quorum of masters is always available.

The article concludes with a summary of the multi‑AZ disaster‑recovery design, practical guidance for upgrading, and a reference to the official Elasticsearch documentation on shard allocation awareness.

Reference : Elasticsearch Cluster Awareness Documentation