Elasticsearch Cluster Architecture and Distributed Data System Design Overview

Distributed systems come in many types, and the design of distributed data systems such as storage, search, and analytics varies greatly. This article focuses on the architecture of Elasticsearch, a widely used open‑source search and analytics platform.

Elasticsearch Cluster Architecture

Elasticsearch is popular for three main scenarios: search (often chosen over Solr), JSON document storage (offering better read/write performance than MongoDB and rich geo/combined queries), and time‑series data analysis (excellent for log processing, monitoring, and visualization).

Key concepts in Elasticsearch:

Node : a running Elasticsearch instance, typically a process on a machine.

Index : a logical collection that includes mapping configuration and inverted/forward index files; an index may span one or many machines.

Shard : a partition of an index that is managed by a node; each shard can be a primary or a replica. Shards of the same index are distributed across different nodes for reliability.

Replica : a copy of a shard that provides strong or eventual consistency.

When creating an index, a document is first routed to the primary shard, indexed there, and then replicated to its replica shards before the operation is considered successful.

Role Deployment Methods

Elasticsearch supports two deployment styles:

Mixed deployment (default): Data and Transport roles coexist on the same node. Requests are sent to any node, which uses a global routing table to forward them to appropriate nodes. This model is simple to start—one node can provide all functions—but request types can interfere with each other, connection limits restrict cluster size, and hot‑updates are not supported.

Tiered deployment : Transport nodes handle request forwarding and result merging, while dedicated Data nodes store and process data. This isolates roles, improves scalability, reduces resource contention, and enables hot‑updates by upgrading Data nodes first, then Transport nodes.

Elasticsearch Data‑Layer Architecture

Indexes and metadata are stored on the local file system using various loading strategies (niofs, mmap, simplefs, smb). By default Elasticsearch selects the optimal method, but users can configure it manually.

Because data resides locally, node failures can cause data loss; replicas mitigate this risk.

Replica Purpose

Setting a replica count of 2 creates three copies of each shard (one primary, two replicas) that are scheduled on different machines or racks. Replicas provide three benefits:

Service availability – traffic can be redirected to remaining replicas if one fails.

Data reliability – loss of a primary does not result in data loss.

Increased query capacity – adding replicas scales read throughput proportionally.

Problems with Replicas

Resource waste – extra replicas consume CPU, memory, and storage without adding processing power when a single shard already meets demand.

Write performance degradation – each write must be applied to the primary and then to all replicas, increasing latency.

Slow scaling – adding replicas requires full data copy, which can be time‑consuming.

Two Types of Distributed Systems

1. Local‑File‑System Based Distributed System

Data is stored on each node’s local disks. When a node fails, the primary shard is elected from a replica, and a new replica is created on another node by copying data from an existing node, which can be slow for large datasets.

2. Distributed‑File‑System (Shared Storage) Based Distributed System

Storage and compute are separated: shards contain only computation logic and reference data stored in a shared distributed file system (e.g., HDFS). This enables elastic scaling of storage and compute independently, reduces waste, and improves hotspot handling, though access latency to the shared storage may be higher.

Conclusion

Both architectures have distinct strengths and weaknesses; choosing the right one depends on specific workload requirements and trade‑offs. Understanding these design choices helps engineers build more reliable and scalable distributed data systems.