Elasticsearch Basics: Core Concepts, Indexing, Write and Search Processes, Cluster Management and Performance Tips

Elasticsearch is a distributed, real‑time full‑text search engine built on Lucene that offers RESTful APIs and indexes every field for fast storage, search, and analytics of massive data sets.

Key concepts include index (similar to a database), type (like a table), document (a record), field , shard (horizontal partition), replica (redundant copy), inverted index, DocValues for sorting/aggregation, the distinction between text (analyzed) and keyword (not analyzed) fields, stop‑word filtering, and the difference between query (scores results) and filter (boolean match only).

The write flow starts with a client sending a request to a coordinating node, which routes the document to the appropriate primary shard; the primary processes the write and replicates it to replica shards, after which the coordinating node returns a response once all required shards succeed.

Internally, documents are first written to a memory buffer, periodically refreshed into a new segment on the filesystem cache. To guarantee durability, the translog records every operation; on failure, Elasticsearch replays the translog. When the translog grows (size or time thresholds), a flush creates a new commit point and clears old translog files.

Updates and deletions are handled by marking the old document as deleted in a .del file and writing a new document; segment merges later purge truly deleted docs.

Search executes in two phases – Query (each shard builds a priority queue of matching docs) and Fetch (the coordinating node retrieves the full documents). An alternative DFS Query Then Fetch performs a pre‑query to collect term and document frequencies for more accurate scoring.

To ensure read‑write consistency under high concurrency, Elasticsearch uses optimistic concurrency control with the _version field, supports write consistency levels ( one, quorum, all), and allows reads to be directed to the primary via the _preference parameter when replication is set to sync or async.

Master node election is managed by the ZenDiscovery module, which relies on discovery.zen.minimum_master_nodes to avoid split‑brain scenarios; nodes vote based on sorted IDs, and a node becomes master only after receiving the required majority of votes.

Performance‑enhancing practices for indexing include using SSDs, batching bulk requests (5‑15 MB), temporarily disabling replicas ( index.number_of_replicas: 0), increasing index.refresh_interval, adjusting segment merge throttling, and raising index.translog.flush_threshold_size.

For deep pagination, Elasticsearch limits results beyond 10 000 records; the recommended approach is to avoid deep pages or use the scroll API, which creates a snapshot ID and iteratively fetches subsequent batches without being affected by ongoing data changes.