Master ELK Stack: From Basics to Advanced Deployment and Sharding Strategies

ELK Stack Overview

ELK is a combination of three open‑source tools—Elasticsearch, Logstash, and Kibana—used for real‑time log collection, storage, search, and visualization.

Flexible processing: Elasticsearch provides real‑time full‑text indexing and powerful search.

Simple configuration: JSON API for Elasticsearch, modular configuration for Logstash, minimal configuration for Kibana.

High retrieval performance: supports billions of documents with second‑level response.

Linear cluster scaling: both Elasticsearch and Logstash can scale horizontally.

Rich front‑end UI: Kibana offers attractive and easy‑to‑use visualizations.

Log Collection Software

ELK Stack

Flume

日志易 (LogEasy)

Component Definitions

Elasticsearch is a highly scalable open‑source full‑text search and analytics engine that stores logs and provides distributed, high‑availability APIs for massive log data such as Nginx, Tomcat, and system logs.

Logstash (or Filebeat) collects and forwards logs, supports plugins for filtering, and can parse plain or custom JSON log formats.

Kibana visualizes data retrieved from Elasticsearch via API calls, offering web‑based graphical log dashboards.

Deploying Elasticsearch

Environment preparation

Hostname

External IP

Internal IP

Role

Applications

ELKstack01

10.0.0.81

172.16.1.81

ES log storage

JDK, elasticsearch

ELKstack02

10.0.0.82

172.16.1.82

ES log storage

JDK, elasticsearch

Installation steps (run as root):

# 1. Replace the official repo
vim /etc/yum.repos.d/es.repo
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

# 2. Install Elasticsearch
yum install -y elasticsearch

# 3. Edit configuration
vim /etc/elasticsearch/elasticsearch.yml
cluster.name: elkstack
node.name: es02
path.data: /data/es/data
path.logs: /data/es/logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.0.0.81","10.0.0.82"]

# 4. Adjust systemd service (remove memory limit comment)
vim /usr/lib/systemd/system/elasticsearch.service
LimitMEMLOCK=infinity

# 5. Create log and data directories
mkdir -p /data/es/{logs,data}

# 6. Set ownership
chown -R elasticsearch.elasticsearch /data/

# 7. Increase file descriptor limits
vim /etc/security/limits.conf
* soft memlock unlimited
* hard memlock unlimited
* soft nofile 131072
* hard nofile 131072

# 8. Set JVM heap
vim /etc/elasticsearch/jvm.options
-Xms1g
-Xmx1g

Access Elasticsearch via browser: http://10.0.0.82:9200/

Installing the Elasticsearch Head Plugin

# 1. Install npm
yum install -y npm

# 2. Clone the repository
git clone https://github.com/mobz/elasticsearch-head.git

# 3. Verify download
ll
# shows elasticsearch-head-master.zip

# 4. Unzip
unzip elasticsearch-head-master.zip

# 5. Build front‑end
cd elasticsearch-head-master
npm install grunt --save

# 6. Start the UI
npm run start &

If you encounter the error below, install bzip2: yum install -y bzip2 After fixing, access the UI at http://10.0.0.81:9100/

Replica Shard Basics

Replica shards provide fault tolerance; if a primary shard fails, a replica is promoted. During indexing, the primary shard writes the document and then synchronizes it to all replicas. Adding replicas improves read throughput but consumes additional hardware resources.

Elasticsearch Working Mechanism

Inverted Index

Unlike traditional databases, Elasticsearch indexes every term in each field, enabling full‑text search. An inverted index maps each term to the documents containing it, along with term frequencies and other statistics used for scoring.

Example with four documents:

Term

txt1

txt2

txt3

txt4

zls

y

y

bgx

y

y

lidao

y

y

oldboy

y

y

y

alex

y

y

Searching a term involves looking up this table to find matching documents. The index also stores document length, term frequencies, and other metrics that influence relevance scoring.

Because the inverted index is immutable, it requires no locking and can be cached in memory, reducing I/O and enabling fast read‑only access.

Segments

Segments break the immutable inverted index into smaller, manageable pieces. New documents are first written to an in‑memory buffer; when the buffer reaches a threshold, it is flushed to disk as a new segment, and a new commit point records all available segments. Searches run across all segments and merge results.

Shard and Replica Configuration Guidance

When creating an index, consider the expected data size. Elasticsearch recommends keeping each shard under ~30 GB. For a 200 GB dataset, 7–8 primary shards are reasonable. Do not over‑allocate shards for future growth; add nodes later if needed.

Typical guidelines:

Allocate 1.5–3× the number of nodes as primary shards (e.g., 3 nodes → up to 9 shards).

Maintain at least one replica per primary shard for high availability.

Increase replicas to improve read performance, but be aware of additional hardware costs.

Cluster Health Monitoring

Cluster health status colors:

Green – all primary and replica shards are allocated.

Yellow – some replica shards are missing.

Red – one or more primary shards are missing.

Simple Python script to check health:

#!/usr/bin/env python
#coding:utf-8
import smtplib, subprocess
clusterip = "10.0.0.81"
obj = subprocess.Popen("curl -sXGET http://"+clusterip+":9200/_cluster/health?pretty=true", shell=True, stdout=subprocess.PIPE)
status = eval(obj.stdout.read()).get("status")
if status == "green":
    print("\033[1;32m 0 \033[0m")
elif status == "yellow":
    print("\033[1;33m 1 \033[0m")
else:
    print("\033[1;31m 2 \033[0m")
# API call example
# curl -s -XGET http://10.0.0.81:9200/_cluster/health?pretty=true

By monitoring the health status, you can quickly detect shard allocation issues and take corrective actions.

Summary

ELK provides a powerful, scalable solution for log ingestion, storage, search, and visualization. Understanding inverted indexes, segments, shard/replica design, and proper cluster monitoring is essential for building a reliable logging infrastructure.