Designing Distributed System Architecture with Elasticsearch

Elasticsearch Cluster Architecture

Elasticsearch is a widely used open‑source search and analytics system. Its core concepts include Node (a running Elasticsearch process), Index (logical configuration and inverted index files), Shard (a partition of an index managed by a node), and Replica (a backup of a shard that ensures strong or eventual consistency).

Shards are distributed across nodes to improve reliability; a primary shard and its replicas are placed on different nodes. When a node fails, the remaining replicas keep the service available, but a failover incurs a short downtime and a risk of data loss during the election.

Index Process

During indexing, a document is routed to the primary shard, indexed there, then sent to each replica shard. The operation returns success only after all replicas have indexed the document.

If a primary or replica shard is lost (e.g., due to machine failure), the missing shard is rebuilt by copying data from other replicas, a process that can temporarily reduce system capacity.

Role Deployment Strategies

Elasticsearch supports two deployment styles:

Mixed deployment (default): Data and transport roles coexist on the same node. This is simple to start—one node can provide all functions—but request types interfere with each other, connection limits (each node keeps 13 connections to every other node) restrict cluster size, and hot updates are not supported.

Layered deployment : Separate Transport Nodes handle request forwarding and result merging, while Data Nodes handle storage and computation. This isolates roles, reduces cross‑role impact, allows larger clusters (Transport nodes connect to many Data nodes), and enables hot updates by upgrading Data nodes first.

Data Layer Architecture

Indexes and metadata are stored on the local file system using loading methods such as niofs, mmap, simplefs, or smb. The mmap method offers the best performance. Because data resides locally, node or disk failures can cause data loss, which is mitigated by using replicas.

Replica configuration determines the number of copies; for example, a replica count of 2 creates one primary shard and two replica shards, each placed on different machines or racks to improve availability, reliability, and query capacity.

Advantages and Drawbacks of Replicas

Improved service availability: if a replica fails, traffic can be redirected to others.

Enhanced data reliability: loss of a primary node does not cause data loss when replicas exist.

Scalable query capacity: adding replicas linearly increases read throughput.

However, replicas increase resource consumption, reduce write performance (writes must be propagated to all replicas), and lengthen recovery time when adding replicas to a hot cluster.

Alternative Distributed System Architectures

The article presents two broader designs:

Local‑file‑system based distributed systems : Each shard stores both data and compute locally. Failure of a node requires copying large amounts of data (e.g., 200 GB over a 1 Gbps network takes ~1600 seconds) and may temporarily halt service if no replica exists.

Shared‑storage based distributed systems (storage‑compute separation): Shards contain only compute logic and reference data stored in a distributed file system such as HDFS. When a node fails, a new compute node can quickly attach to the existing data without massive data transfer, offering elastic scaling and better hotspot handling, though access latency to the shared storage may be higher.

Conclusion

Both architectures have distinct strengths and weaknesses; choosing the right one depends on workload characteristics, reliability requirements, and cost considerations. Understanding Elasticsearch’s shard‑replica model and deployment options helps engineers design robust, scalable distributed data systems.