Understanding Elasticsearch: Core Principles and Architecture

1. Elasticsearch Overall Structure

After visualizing the overall principle, we outline Elasticsearch’s architecture.

In a cluster, an index consists of multiple nodes, each node being an Elasticsearch instance.

Each node holds several shards; primary shards (P1, P2) and replica shards (R1, R2).

Every shard corresponds to a Lucene index (the low‑level index files).

A Lucene index is composed of multiple segments (the inverted index files), each storing documents (Docs).

A commit point records information about all segments.

2. Lucene Index Structure

What files appear in the Lucene index diagram?

(More file types can be referenced elsewhere.)

File relationships are shown below:

3. Lucene Processing Flow

Understanding the Lucene processing flow helps you index and search documents more effectively.

Index creation process:

Prepare the original documents; sources may be files, databases, or network streams.

Pass the document content through an analysis component to produce a series of terms.

The indexing component builds a dictionary and an inverted table from the document and its terms.

Search process:

Tokenize the query string into a series of terms.

Use the inverted index to find documents containing those terms and merge them into a result set.

Score each document against the query and return results ordered by relevance.

4. Elasticsearch Analyzers

Syntax analysis and language processing are crucial, so we add knowledge about Elasticsearch analyzers.

The analysis consists of the following steps:

First, split a block of text into independent tokens suitable for an inverted index.

Then, normalize these tokens into a standard form to improve searchability (recall).

An analyzer packages three functions:

Character filter : The string passes sequentially through each character filter, which can, for example, strip HTML or convert '&' to 'and' before tokenization.

Tokenizer : The string is split into individual tokens; a simple tokenizer separates on spaces and punctuation.

Token filter : Tokens pass through each token filter, which may modify tokens (e.g., lowercase "Quick"), delete stop words (e.g., "a", "and", "the"), or add synonyms (e.g., "jump" ↔ "leap").

Elasticsearch ships with ready‑to‑use character filters, tokenizers, and token filters that can be combined into custom analyzers for various purposes.

Built‑in Analyzers

Elasticsearch provides several pre‑packaged analyzers. The following example shows how each analyzer tokenizes the same input string:

"Set the shape to semi-transparent by calling set_trans(5)"

Standard Analyzer

The standard analyzer is the default. It follows Unicode word boundaries, removes most punctuation, and lower‑cases tokens.

set, the, shape, to, semi, transparent, by, calling, set_trans, 5

Simple Analyzer

The simple analyzer splits on any non‑letter character and lower‑cases tokens.

set, the, shape, to, semi, transparent, by, calling, set, trans

Whitespace Analyzer

The whitespace analyzer splits only on spaces.

Set, the, shape, to, semi-transparent, by, calling, set_trans(5)

Language Analyzer (English)

Language‑specific analyzers apply language rules, such as removing stop words and stemming. set, shape, semi, transpar, call, set_tran, 5 Note that "transparent", "calling" and "set_trans" are reduced to their stems.

When to Use Analyzers

When indexing a document, its full‑text fields are analyzed into tokens to build the inverted index. When searching those fields, the query string must undergo the same analysis to ensure token formats match.

Full‑text queries require understanding how each field is defined:

For a full‑text field, the same analyzer is applied to the query string to produce the correct token list.

For an exact‑value field, the query string is not analyzed; the exact value is matched.

Example

Assume one document per day in Elasticsearch. Queries:

GET /_search?q=2014              # 12 results
GET /_search?q=2014-09-15        # 12 results !
GET /_search?q=date:2014-09-15   # 1 result
GET /_search?q=date:2014         # 0 results !

Why the different results?

The date field stores an exact token "2014-09-15".

The _all field is full‑text; the date is tokenized into three tokens: 2014, 09, 15.

Querying _all for 2014 matches all 12 documents because each contains the token 2014: GET /_search?q=2014 # 12 results Querying _all for 2014-09-15 first analyzes the string, producing a query that matches any of the tokens 2014, 09, or 15, again returning all 12 documents: GET /_search?q=2014-09-15 # 12 results ! Querying the date field for 2014-09-15 looks for the exact date token and returns a single document: GET /_search?q=date:2014-09-15 # 1 result Querying the date field for 2014 finds no documents because no exact token "2014" exists:

GET /_search?q=date:2014         # 0 results !