Elasticsearch Overview: Architecture, Core Concepts, and Performance Optimization

Elasticsearch is an open‑source, distributed search and analytics engine built on Apache Lucene, designed to handle both structured and unstructured data at scale.

Data Types – Data in daily life can be classified as structured (e.g., relational tables) or unstructured (e.g., documents, images). Correspondingly, search can be performed via structured data search or full‑text search.

Lucene – Lucene provides the core inverted index and term dictionary needed for full‑text search. Elasticsearch wraps Lucene to expose a simple RESTful API while adding distributed capabilities.

Cluster Basics – A cluster consists of one or more nodes sharing the same cluster.name. Nodes can be master‑eligible, data, or coordinating nodes. Master nodes manage cluster state, shard allocation, and node discovery (Zen Discovery). Data nodes store shards and handle indexing/search operations. Any node can act as a coordinating node to receive client requests.

Node Discovery – Nodes discover each other via unicast hosts defined in discovery.zen.ping.unicast.hosts. The master election follows a deterministic rule based on node IDs, with discovery.zen.minimum_master_nodes preventing split‑brain scenarios.

Shards and Replicas – An index is divided into a fixed number of primary shards ( number_of_shards) and optional replica shards ( number_of_replicas). Shards enable horizontal scaling; replicas provide high availability and load balancing. Shard allocation follows the formula shard = hash(routing) % number_of_primary_shards, where routing defaults to the document _id.

Mapping – Mapping defines field types (e.g., text, keyword, integer, date) and indexing behavior. Dynamic mapping infers types automatically, while explicit mapping allows precise control over analyzers and storage.

Installation & Basic Usage – Download and unzip Elasticsearch; start with bin/elasticsearch. The service listens on port 9200, exposing cluster information via GET /. Index creation, shard/replica settings, and mappings are performed with JSON payloads, e.g.:

PUT my_index
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  },
  "mappings": {
    "_doc": {
      "properties": {
        "title": {"type": "text"},
        "age": {"type": "integer"},
        "created": {"type": "date", "format": "strict_date_optional_time||epoch_millis"}
      }
    }
  }
}

Cluster Health – Check health with GET /_cluster/health, which returns status green, yellow, or red indicating shard allocation and replica availability.

Indexing Workflow – Documents are first written to the JVM heap and recorded in the transaction log (translog). Periodic refresh (default 1 s) creates a new segment in the file system cache, making the data searchable. When the translog reaches 512 MB or 30 min, a flush persists data to disk and clears the translog.

Storage Model – Elasticsearch stores data in immutable segments. New documents create new segments; deletions are recorded in .del files, and updates are treated as delete + add. Segment merging runs in the background to consolidate segments, reclaim space, and improve search performance.

Performance Optimizations – Use SSDs or RAID 0 for fast I/O, avoid remote mounts, and configure multiple path.data directories. Optimize mappings (use keyword instead of text when appropriate), disable unnecessary doc values, and tune index.refresh_interval and replica count during bulk indexing. Prefer scroll APIs over deep pagination, and set deterministic routing values to target specific shards.

JVM Tuning – Set Xms and Xmx to the same value (≤ 50 % of physical RAM, max 32 GB). Consider G1GC, ensure sufficient file‑system cache, and monitor heap usage.