ELK Stack Installation and Configuration Guide for macOS and Linux

When dealing with large‑scale websites, the author became interested in middleware technology and recorded the configuration process for the ELK stack (Elasticsearch + Logstash + Kibana) to collect, analyze, and visualize logs.

Why Use ELK

ELK consists of three tools—Elasticsearch, Logstash, and Kibana. It enables log collection, analysis, and visual presentation, which becomes essential as business volume and data size grow beyond simple file‑based logging.

Architecture Diagram

Environment Preparation

Both macOS and Linux setups assume JDK 1.8+ is installed.

Elasticsearch

Elasticsearch is a distributed RESTful search and analytics engine, the core of the Elastic Stack.

Mac installation:

brew install elasticsearch
elasticsearch

Linux installation (example for version 6.2.4):

# curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz
# tar -zxvf elasticsearch-6.2.4.tar.gz
# cd elasticsearch-6.2.4
# ./bin/elasticsearch

Note: On Linux a dedicated user/group is required; see the troubleshooting section for permission issues.

Logstash

Logstash is an open‑source data‑processing pipeline that can ingest data from multiple sources, transform it, and send it to a storage backend (Elasticsearch).

Installation:

# macOS
brew install logstash

# Linux
# curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz
# tar -zxvf logstash-6.3.2.tar.gz

Configuration (logstash.conf):

input { stdin { } }
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

Run Logstash:

bin/logstash -f logstash.conf

Kibana

Kibana provides a UI for visualizing data stored in Elasticsearch.

Installation:

# macOS
brew install kibana

# Linux
# curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz
# tar -zxvf kibana-6.3.2-linux-x86_64.tar.gz

Start Kibana:

./bin/kibana

Integrating Spring + Logstash

Add Logstash Logback encoder dependency:

<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>5.2</version>
</dependency>

Configure Logback (logback.xml) to send logs to Logstash:

<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
  <destination>localhost:9250</destination>
  <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder">
    <customFields>{"appname":"ye_test"}</customFields>
  </encoder>
</appender>

<root level="INFO">
  <appender-ref ref="stash"/>
</root>

Troubleshooting (Pitfalls)

Memory Allocation Errors

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(...) failed; error='Cannot allocate memory' (errno=12)

Solution: Reduce JVM heap size in jvm.options (e.g., -Xms256m and -Xmx256m).

Running Elasticsearch as Root

org.elasticsearch.bootstrap.StartupException: can not run elasticsearch as root

Create a non‑root user and adjust ownership:

# groupadd es
# useradd es -g es -p es
# chown -R es:es /path/to/elasticsearch
# sudo su - es

Virtual Memory Limit (vm.max_map_count)

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Increase the limit:

# vim /etc/sysctl.conf   # add: vm.max_map_count=655360
# sysctl -p

References

ELK Quick Guide

Setting up ELK on macOS

Building a centralized log platform with ELK

Logback JSON encoder

Common Elasticsearch installation errors

Linux root user missing JAVA_HOME

Remote connection issues with Elasticsearch

For further learning, the author invites readers to join the architecture community and share the article.