A Comprehensive Introduction to Elasticsearch: Architecture, Core Concepts, and Practical Usage

Elasticsearch is an open‑source, distributed, near‑real‑time search and analytics engine built on top of Apache Lucene.

It handles both structured (relational) and unstructured (full‑text) data by creating inverted indexes that map terms to the documents in which they appear.

Lucene provides the core indexing and search capabilities; Elasticsearch adds clustering, RESTful APIs, and distributed features such as automatic node discovery (Zen Discovery) and master election.

Key concepts include clusters, nodes (master, data, coordinating), shard allocation, replica shards for high availability, and the routing formula shard = hash(routing) % number_of_primary_shards that determines which primary shard stores a document.

Indices are created with a fixed number of primary shards and optional replicas; mappings define field types (text, keyword, date, etc.) and can be static or dynamic.

Installation is straightforward: download, unzip, and run bin/elasticsearch. The default HTTP port is 9200, and a simple curl http://localhost:9200 returns cluster information.

Cluster health is reported as green, yellow, or red, indicating the status of primary and replica shards.

Write operations first go to an in‑memory buffer, are recorded in the transaction log, and become visible after a refresh (default 1 s). A flush creates a commit point, persists data to disk, and clears the translog.

Data is stored on disk as immutable segments; deletions and updates are handled via delete markers and new segments. Periodic segment merging reduces the number of segments, reclaims space, and improves query performance.

Performance can be improved by using SSDs, RAID 0, multiple data paths, appropriate shard counts, disabling unnecessary doc values, using keyword fields instead of text where possible, adjusting index.refresh_interval, and tuning JVM heap size and garbage collection settings.