Elasticsearch Fundamentals: Architecture, Indexing, Sharding, and Performance Optimization

Elasticsearch is an open‑source, distributed search and analytics engine built on top of Apache Lucene. It stores data in indices that are divided into primary shards and replica shards, allowing horizontal scaling and high availability.

Data is routed to a specific primary shard using the formula shard = hash(routing) % number_of_primary_shards, where the routing value defaults to the document’s _id but can be customized. The coordinating node calculates the target shard and forwards the write request to the primary shard, which then replicates the operation to its replicas.

Internally, Elasticsearch writes documents to an in‑memory buffer and records the operation in a transaction log (translog). Periodically (default every 1 second) a refresh creates a new immutable segment in the file system cache, making the data searchable. When the translog reaches 512 MB or 30 minutes, a flush persists the segment to disk, creates a commit point, and clears the translog.

Segments are immutable; updates and deletions are handled by writing new segments and marking old documents as deleted in a .del file. Over time, many small segments would degrade query performance, so Elasticsearch runs background segment merges that combine small segments into larger ones, discarding deleted documents in the process.

Key configuration examples:

PUT /my_index
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1,
    "refresh_interval": "30s"
  },
  "mappings": {
    "properties": {
      "title": { "type": "text" },
      "age": { "type": "integer" },
      "created": { "type": "date", "format": "strict_date_optional_time||epoch_millis" }
    }
  }
}

Cluster discovery (Zen Discovery) can be tuned with:

discovery.zen.ping.unicast.hosts: ["host1", "host2:9300"]
# Minimum master nodes to avoid split‑brain
discovery.zen.minimum_master_nodes: 2

Node roles are defined in elasticsearch.yml:

node.master: true   # eligible to be master
node.data: true     # stores data and handles queries

Performance recommendations include using SSDs or RAID‑0, allocating multiple data paths, disabling unnecessary doc_values, using keyword fields when full‑text search isn’t needed, increasing refresh_interval for bulk indexing, and employing scroll APIs for deep pagination.

JVM tuning is also crucial: set the heap size with identical -Xms and -Xmx values (typically ≤ 50 % of physical RAM), consider the G1 garbage collector, and ensure sufficient file‑system cache for fast segment reads.