Elasticsearch Cluster Core Principles & Optimization Guide

2.1 System Architecture

A running Elasticsearch instance is called a node, and a cluster consists of one or more nodes sharing the same cluster.name configuration. Together they bear the data and load pressure. When a node joins or leaves the cluster, the cluster rebalances all data automatically.

When a node is elected as the master node, it manages cluster-wide changes such as creating or deleting indices, or adding or removing nodes. If a node serves as both master and data node, it handles document-level changes and searches, as well as metadata management tasks.

As a user, you can send requests to any node in the cluster, including the master node. Every node knows the location of any document and can forward the request directly to the node storing the required document. Regardless of which node receives the request, it collects data from all nodes containing the needed documents and returns the final result to the client. Elasticsearch manages all of this transparently.

2.2 Read/Write Overview and Optimization

1. Read/Write Request Workflow

1.1 Write Process

Client → Coordinating Node → Primary Shard → Replica Shards → Response

Coordinating Node : Calculates the target primary shard location based on the routing parameter (default document ID).

Primary Shard Processing :

Replica Sync : Primary shard replicates in parallel to all replica shards.

Response Strategy :

PUT /my_index/_doc/1?wait_for_active_shards=all  # Wait for all replicas to be ready

1.2 Read Process

Client → Coordinating Node → Any Shard (Primary/Replica) → Merge Results → Response

Request Routing : Coordinating node randomly selects a healthy shard.

Load Balancing : Automatically distributes requests across all available shards.

Consistency Guarantee :

# Force refresh to ensure visibility
GET /my_index/_doc/1?refresh=true

2. Key Mechanisms for Data Ingestion

2.1 Bulk Write (Bulk API)

# Python example: using Elasticsearch-py for bulk writes
from elasticsearch import Elasticsearch

es = Elasticsearch()
bulk_data = []

for doc in documents:
    bulk_data.append({
        'index': {
            '_index': 'my_index',
            '_id': doc['id']
        }
    })
    bulk_data.append(doc)

es.bulk(body=bulk_data)

Performance Optimization :

2.2 Async Write

# Extend refresh interval to improve write performance
PUT /my_index/_settings
{
  "index.refresh_interval": "30s"
}

2.3 Data Streams

# Create a data stream
PUT _index_template/my_template
{
  "index_patterns": ["my-data-stream-*"],
  "data_stream": {}
}

# Write data
POST my-data-stream/_doc
{
  "@timestamp": "2023-07-16T12:00:00",
  "message": "Hello World"
}

Automatically manages underlying index rollover.

Supports efficient time-series data writes.

3. High Availability Design and Implementation

1. Shard Allocation Strategy

# Configure shard allocation awareness
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "zone"
  }
}

Ensures primary and replica shards of the same shard are distributed across different availability zones.

Use _cat/shards to monitor shard distribution.

2. Failure Detection and Recovery

# View recovering shards
GET _cat/recovery?v

Master node monitors cluster state and detects node failures.

Automatically reallocates unavailable shards to healthy nodes.

3. Cross-Cluster Replication (CCR)

# Configure remote cluster
PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.production_cluster.seeds": [
      "prod-node1:9300",
      "prod-node2:9300"
    ]
  }
}

# Create follower index
PUT books-follower
{
  "settings": {
    "index.follow.remote_cluster": "production_cluster",
    "index.follow.leader_index": "books"
  }
}

4. Data Skew Diagnosis and Handling

1. Detecting Data Skew

# View shard sizes
GET _cat/shards?v&s=store:desc

# View node load
GET _nodes/stats/process,thread_pool

Shard size difference exceeding 20% is considered skew.

Severe imbalance in node CPU/memory usage.

2. Common Skew Causes

3. Solutions

# Example: using custom routing key
POST /my_index/_doc/1?routing=user123
{
  "user_id": "user123",
  "message": "Hello"
}

# Example: three data nodes, three shards, limit two shards per node (primary+replica)
PUT your_index_name/_settings
{
  "index.routing.allocation.total_shards_per_node": 2
}

Composite Routing Key : routing=user_id%shard_count Index Rollover : Create new indices by time (e.g., daily_index-2023-07-16).

Forced Balancing :

POST /_cluster/reroute?retry_failed=true

5. Performance Optimization Best Practices

1. Write Optimization

# Bulk write configuration
PUT /my_index/_settings
{
  "index.refresh_interval": "30s",
  "index.translog.durability": "async",
  "index.number_of_replicas": 0  # Temporarily disable replicas during import
}

2. Read Optimization

# Use filter queries
GET /my_index/_search
{
  "query": {
    "bool": {
      "filter": [
        {"term": {"status": "active"}},
        {"range": {"timestamp": {"gte": "now-7d"}}}
      ]
    }
  }
}

3. Cluster Configuration

# elasticsearch.yml optimized settings
bootstrap.memory_lock: true
network.host: 0.0.0.0
discovery.seed_hosts: ["node1", "node2", "node3"]
cluster.initial_master_nodes: ["node1"]

6. Summary

By deeply understanding these principles and combining performance optimization best practices, you can build high-performance, highly available Elasticsearch clusters. Regular stress testing and capacity planning are recommended to ensure cluster stability as business grows.