Mastering ELK: A Step‑by‑Step Guide to Install Elasticsearch, Logstash & Kibana

Why use ELK

ELK consists of three tools—Elasticsearch, Logstash, and Kibana—used to collect, analyze, and visualize logs. As business traffic grows, simple file logging and grep become insufficient, so ELK handles large‑scale log ingestion and analysis.

Simple architecture diagram

Environment configuration

Both macOS and Linux require JDK 1.8 or higher.

[root@VM_234_23_centos ~]# java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

Note: ELK 6.0+ also requires JDK 1.8 or newer.

Elasticsearch

Elasticsearch is a distributed RESTful search and analytics engine, the core of the Elastic Stack.

macOS installation:

brew install elasticsearch
elasticsearch

Linux installation:

# curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz
# tar -zxvf elasticsearch-6.2.4.tar.gz
# cd elasticsearch-6.2.4
# ./bin/elasticsearch

On Linux you need a dedicated user/group for Elasticsearch; see the troubleshooting section for details.

Logstash

Logstash is an open‑source data‑processing pipeline that can ingest data from multiple sources, transform it, and send it to a destination such as Elasticsearch.

Installation:

# macOS
brew install logstash

# Linux
curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz
tar -zxvf logstash-6.3.2.tar.gz

Configuration (logstash.conf):

input { stdin { } tcp { host => "127.0.0.1" port => 9250 mode => "server" tags => ["tags"] codec => json_lines } }
output { elasticsearch { hosts => ["localhost:9200"] } stdout { codec => rubydebug } }

Run Logstash: bin/logstash -f logstash.conf Verify by accessing http://localhost:9600/ which returns JSON confirming the service is up.

Common pitfall: insufficient memory on low‑spec servers. Reduce JVM heap in config/jvm.options (e.g., -Xms256m and -Xmx256m ) to avoid “Cannot allocate memory” errors.

Kibana

Kibana visualizes data stored in Elasticsearch, allowing you to explore logs and create dashboards.

Installation:

# macOS
brew install kibana

# Linux
curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz
tar -zxvf kibana-6.3.2-linux-x86_64.tar.gz

Configuration (optional): edit config/kibana.yml to set elasticsearch.url if you changed defaults.

Run Kibana: ./bin/kibana Access the UI at http://localhost:5601/app/kibana#/home and add the logstash-* index pattern to start visualizing logs.

Integrating Spring + Logstash

1. Update logstash.conf to listen on a TCP port (as shown above).

2. Add Maven dependency:

<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>5.2</version>
</dependency>

3. Configure logback.xml to send logs to Logstash:

<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
  <destination>localhost:9250</destination>
  <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder">
    <customFields>{"appname":"ye_test"}</customFields>
  </encoder>
</appender>

<root level="INFO">
  <appender-ref ref="stash"/>
</root>

Logstash will automatically create an index (e.g., logstash-2021.08.16) when it receives logs.

Pitfalls & Troubleshooting

Startup errors

Running Elasticsearch as root results in org.elasticsearch.bootstrap.StartupException: can not run elasticsearch as root . Create a non‑root user and adjust file ownership.

# groupadd es
# useradd es -g es -p es
# chown -R es:es /path/to/elasticsearch
# sudo su - es

Permission errors may also occur; ensure the user has read/write access to the installation directory.

Memory‑related failures

“Killed” messages indicate the JVM ran out of memory. Reduce heap size in jvm.options (e.g., -Xms512m , -Xmx512m ).

Virtual memory limits

Elasticsearch requires vm.max_map_count ≥ 262144. Increase it by adding vm.max_map_count=655360 to /etc/sysctl.conf and running sysctl -p , then restart Elasticsearch.