Step‑by‑Step Guide to Installing and Configuring the ELK Stack (Elasticsearch, Logstash, Kibana) on macOS and Linux

Why use ELK – ELK (Elasticsearch + Logstash + Kibana) enables centralized log collection, powerful search, and visual dashboards, which become essential as application traffic and log volume grow beyond simple file‑based logging.

Environment preparation – Ensure JDK 1.8+ is installed on the host (macOS or Linux). Verify the Java version with:

[root@VM_234_23_centos ~]# java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

Elasticsearch installation

macOS: brew install elasticsearch then run elasticsearch Linux: download, extract and start:

# curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz
# tar -zxvf elasticsearch-6.2.4.tar.gz
# cd elasticsearch-6.2.4
# ./bin/elasticsearch

Logstash installation and configuration

macOS: brew install logstash Linux: download and extract:

# curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz
# tar -zxvf logstash-6.3.2.tar.gz

Create a simple logstash.conf (or logstash.conf) file:

input { stdin { } }
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

Run Logstash with the configuration: bin/logstash -f logstash.conf Verify the pipeline by accessing http://localhost:9600/ and checking the Elasticsearch logs for a line similar to:

[2018-08-16T14:08:36,436][INFO][o.e.c.m.MetaDataIndexTemplateService] [f2s1SD8] adding template [logstash] for index patterns [logstash-*]

Kibana installation and configuration

macOS: brew install kibana Linux: download and extract:

# curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz
# tar -zxvf kibana-6.3.2-linux-x86_64.tar.gz

Start Kibana: # ./bin/kibana Open http://localhost:5601/app/kibana#/home to view the UI, add an index pattern (e.g., logstash-*), and explore the Discover and Visualize sections.

Integrating Spring applications with Logstash

Add the Logstash Logback encoder dependency:

<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>5.2</version>
</dependency>

Configure logback.xml to send JSON logs to Logstash:

<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
  <destination>localhost:9250</destination>
  <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder">
    <customFields>{"appname":"ye_test"}</customFields>
  </encoder>
</appender>

<root level="INFO">
  <appender-ref ref="stash"/>
</root>

Update Logstash to listen on the same port:

input {
  tcp {
    host => "127.0.0.1"
    port => 9250
    mode => "server"
    codec => json_lines
  }
}
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

Common pitfalls and fixes

Running as root – Elasticsearch refuses to start as root. Create a dedicated user (e.g., es) and adjust ownership of the installation directory.

Insufficient JVM heap – Reduce heap size in config/jvm.options:

-Xms256m
-Xmx256m

Virtual memory limits – Increase vm.max_map_count:

# echo "vm.max_map_count=655360" >> /etc/sysctl.conf
# sysctl -p

After applying these changes, restart the services and verify that logs flow from the Spring application through Logstash into Elasticsearch and become visible in Kibana dashboards.