Master Elastic Stack: Install and Configure Elasticsearch, Logstash, Kibana, Filebeat

About Elastic Stack

Elastic Stack (formerly ELK Stack) consists of Elasticsearch, Logstash, Kibana, and Filebeat, forming a practical and easy‑to‑use monitoring architecture for visualizing massive log data.

Elasticsearch Overview

Elasticsearch is a highly scalable open‑source full‑text search and analytics engine that stores, searches, and analyzes large volumes of data in near real‑time, serving as the underlying engine for applications requiring complex search capabilities.

Logstash Overview

Logstash is a lightweight open‑source log‑collection engine that ingests, enriches, and forwards data. It works like a pipeline with input, filter, and output stages, often sending processed logs to Elasticsearch.

Kibana Overview

Kibana is an open‑source analytics and visualization platform that works with Elasticsearch, allowing users to create dynamic dashboards, charts, tables, and maps via a browser‑based interface.

Filebeat Overview

Filebeat, part of the Beats family, is a lightweight log shipper installed on hosts to forward various log types (e.g., Windows event logs, system metrics) to the ELK stack. It complements Logstash rather than replacing it.

Elastic Stack Installation & Configuration

Logstash Installation

1. Install Logstash (requires Java 8, not Java 9). yum install java 2. Import the public signing key.

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

3. Add the Logstash YUM repository. vim /etc/yum.repos.d/logstash.repo 4. Install Logstash.

yum install logstash

Elasticsearch Installation

1. Download the tarball.

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz

2. Extract the package. tar -xzvf elasticsearch-6.4.0.tar.gz 3. Edit config/elasticsearch.yml (example settings):

cluster.name: my-elk

node.name: es1

path.data: /elasticsearch/elasticsearch-6.4.0/data

path.logs: /elasticsearch/elasticsearch-6.4.0/logs/

network.host: 192.168.179.134

http.port: 9200

4. Start Elasticsearch.

cd elasticsearch-6.4.0/

./bin/elasticsearch

Run as a regular user and adjust directory permissions if needed.

5. Verify the service (e.g., netstat -ntlp | grep 9200 or curl 192.168.179.134:9200).

Common startup warnings and fixes:

Increase file descriptor limit to at least 65536 (edit /etc/security/limits.conf).

Increase vm.max_map_count to at least 262144 (edit /etc/sysctl.conf and run sysctl -p).

Elasticsearch Head Plugin Installation

1. Install Node.js (source compilation shown).

wget https://nodejs.org/dist/v8.11.4/node-v8.11.4.tar.gz

tar -zxvf node-v8.11.4.tar.gz

./configure && make && make install

2. Install Grunt CLI. npm install -g grunt-cli 3. Clone and set up the head plugin.

git clone https://github.com/mobz/elasticsearch-head.git

cd elasticsearch-head/

npm install

npm install [email protected] --ignore-scripts

4. Enable CORS in elasticsearch.yml:

http.cors.enabled: true

http.cors.allow-origin: "*"

5. Start the plugin and open in a browser. npm run start Access via http://192.168.179.134:9100/.

Kibana Installation

1. Download and extract Kibana.

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz

tar -xzvf kibana-6.4.0-linux-x86_64.tar.gz

2. Edit kibana.yml (example settings):

server.port: 5601

server.host: "192.168.179.134"

elasticsearch.url: "http://192.168.179.134:9200"

kibana.index: ".kibana"

3. Start Kibana. /kibana-6.4.0-linux-x86_64/bin/kibana Verify with netstat -ntlp | grep 5601 and open http://192.168.179.134:5601 in a browser.

Typical warnings involve missing security keys; add appropriate xpack.security.encryptionKey entries to kibana.yml to silence them.

Key Elasticsearch Concepts & Operations

Cluster

A cluster is a collection of one or more nodes that store data and provide unified indexing and search across all nodes.

Cluster Health

Health status is reported as green, yellow, or red indicating the state of primary and replica shards. green: all primary and replica shards are active. yellow: all primary shards are active, but some replicas are not. red: one or more primary shards are missing.

Check health via curl -X GET "192.168.179.134:9200/_cluster/health" or via browser.

Node

A node is a single server that stores data and participates in indexing and search. One node can be elected master to manage cluster‑wide changes.

Index

An index is a logical namespace that points to one or more physical shards, grouping documents with similar characteristics.

Shard

A shard is a low‑level unit that holds a subset of the data. Elasticsearch distributes shards across nodes for scalability and redundancy. Shards can be primary or replica; replicas provide fault tolerance and improve read performance.

Index vs. Shard

In Elasticsearch, a Lucene index corresponds to a shard. An Elasticsearch index is a collection of shards; queries are sent to each shard and results are merged.

Creating an Index

Example command to create an index with 3 primary shards and 1 replica:

curl -X PUT "192.168.179.134:9200/blogs" -H 'Content-Type: application/json' -d '{"settings": {"number_of_shards": 3, "number_of_replicas": 1}}'

List indices with curl -X GET "http://192.168.179.134:9200/_cat/indices?v".

Creating a Document

Example:

curl -X PUT "192.168.179.134:9200/customer/_doc/1?pretty" -H 'Content-Type: application/json' -d '{"name": "John Doe"}'

Retrieve with curl -X GET "192.168.179.134:9200/customer/_doc/1?pretty".

Deleting an Index

Remove a specific index: curl -X DELETE "192.168.179.134:9200/customer" Delete all indices:

curl -X DELETE "192.168.179.134:9200/_all"

Deleting a Document

Example: curl -X DELETE "192.168.179.134:9200/customer/_doc/1?pretty" The basic REST pattern is <METHOD> /<index>/<type>/<id>.