Mastering ELK Stack: Step‑by‑Step Guide to Deploy Elasticsearch, Logstash, Kibana, and Filebeat

Introduction

Recently I have been tinkering with the ELK log platform, a complete solution from Elastic for log collection, analysis, and visualization.

After hands‑on practice I discovered many steps and pitfalls, which I record here.

Highlights : step‑by‑step screenshots, real‑world cases, pitfall notes, integration with development logs.

Log collection architectures

ELK Stack: Elasticsearch + Logstash + Filebeat + Kibana – the most common architecture.

Elasticsearch + Logstash + Kafka + Kibana – adds a message broker, covered in a later tutorial.

This article focuses on the ELK Stack, which is non‑intrusive to our code: it collects disk log files and imports them into Elasticsearch.

For example, an application writes logs via Logback to disk, and the middleware stack gathers them for querying.

The overall architecture diagram is shown below.

Process flow

Filebeat collects logs and forwards them to Logstash.

Logstash performs powerful data cleansing.

The data is finally written to Elasticsearch.

Kibana visualizes the data.

All examples were performed on a single Ubuntu VM with 6 GB memory.

1. Deploy Elasticsearch

Pull the Elasticsearch image docker pull elasticsearch:7.7.1 Create mount directories mkdir -p /data/elk/es/{config,data,logs} Set permissions chown -R 1000:1000 /data/elk/es Create configuration file

cd /data/elk/es/config
touch elasticsearch.yml
# configuration content
cluster.name: "my-es"
network.host: 0.0.0.0
http.port: 9200

Run the Elasticsearch container

docker run -it -d -p 9200:9200 -p 9300:9300 --name es -e ES_JAVA_OPTS="-Xms1g -Xmx1g" -e "discovery.type=single-node" --restart=always -v /data/elk/es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /data/elk/es/data:/usr/share/elasticsearch/data -v /data/elk/es/logs:/usr/share/elasticsearch/logs elasticsearch:7.7.1

Verify Elasticsearch is up

curl http://localhost:9200

2. Deploy Kibana

2.1 Install Kibana

Pull the Kibana image docker pull kibana:7.7.1 Obtain Elasticsearch container IP

docker inspect --format '{{ .NetworkSettings.IPAddress }}' es

Result: 172.17.0.2

Create Kibana configuration file

mkdir -p /data/elk/kibana/
vim /data/elk/kibana/kibana.yml

Configuration content

# Default Kibana configuration for Docker
server.name: kibana
server.host: "0"
elasticsearch.hosts: ["http://172.17.0.2:9200"]
xpack.monitoring.ui.container.elasticsearch.enabled: true

2.2 Run Kibana

docker run -d --restart=always --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 --name kibana -p 5601:5601 -v /data/elk/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml kibana:7.7.1

Access http:// :5601. Kibana will prompt to load sample data.

After importing sample data you can use the Discover view to search logs.

3. Deploy Logstash

3.1 Install Java JDK

sudo apt install openjdk-8-jdk

Modify /etc/profile to set JAVA_HOME and PATH, then source it.

# JAVA
JAVA_HOME="/usr/lib/jdk/jdk-12"
PATH="$PATH:$JAVA_HOME/bin"
source /etc/profile
java -version

3.2 Install Logstash

curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-7.7.1.tar.gz
tar -xzvf logstash-7.7.1.tar.gz
cd logstash-7.7.1
bin/logstash -e 'input { stdin { } } output { stdout {} }'

Enter a line of text; the output appears on stdout, confirming a successful installation.

Enable automatic config reload in config/logstash.yml by setting config.reload.automatic: true.

3.3 Configure Logstash

Logstash requires at least an input and an output, with optional filter sections.

Create weblog.conf:

mkdir -p /logstash-7.7.1/streamconf
vim /logstash-7.7.1/streamconf/weblog.conf

Configuration content:

input {
  tcp {
    port => 9900
  }
}
filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  mutate {
    convert => { "bytes" => "integer" }
  }
  geoip {
    source => "clientip"
  }
  useragent {
    source => "agent"
    target => "useragent"
  }
  date {
    match => ["timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
  }
}
output {
  stdout { }
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

Test the pipeline: head -n 1 weblog-sample.log | nc localhost 9900 The log line appears in the console, confirming the pipeline works.

Search the inserted document in Elasticsearch:

GET logstash/_search

4. Deploy Filebeat

4.1 Install Filebeat

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.1-linux-x86_64.tar.gz
tar xzvf filebeat-7.7.1-linux-x86_64.tar.gz

Note: replace version 7.7.1 with the desired one.

4.2 Configure Filebeat

Create filebeat_apache.yml to send logs directly to Elasticsearch for a quick test:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/vagrant/logs/*.log

output.elasticsearch:
  hosts: ["192.168.56.10:9200"]

Adjust paths to match your log directory.

4.3 Test Filebeat

Start Logstash first, then Filebeat:

bin/logstash -f weblog.conf
./filebeat -e -c filebeat_apache.yml

Filebeat should report that it is monitoring three files: error.log, info.log, debug.log.

Verify the indices in Elasticsearch:

curl http://localhost:9200/_cat/indices?v

Create an index pattern in Kibana (e.g., filebeat-*) and use the Discover view to browse the imported logs.

4.4 Filebeat + Logstash

Configure Filebeat to forward logs to Logstash, which then writes to Elasticsearch:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/vagrant/logs/*.log

output.logstash:
  hosts: ["localhost:9900"]

Update weblog.conf to use a beats input and adjust the useragent field:

input {
  beats {
    port => "9900"
  }
}
filter {
  grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
  mutate { convert => { "bytes" => "integer" } }
  geoip { source => "clientip" }
  useragent { source => "user_agent" target => "useragent" }
  date { match => ["timestamp","dd/MMM/yyyy:HH:mm:ss Z"] }
}
output {
  stdout { codec => dots {} }
  elasticsearch {
    hosts => ["192.168.56.10:9200"]
    index => "apache_elastic_example"
  }
}

After restarting both services, the logs appear in the apache_elastic_example index.

Ensure the Grok pattern matches your log format; otherwise fields will not be parsed correctly.

5. Troubleshooting

5.1 Kibana image pull fails (out of memory)

Cause: exhausted inodes. Clean up files and run:

df -i
sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
curl -s https://raw.githubusercontent.com/ZZROTDesign/docker-clean/v2.0.4/docker-clean | sudo tee /usr/local/bin/docker-clean > /dev/null && \
sudo chmod +x /usr/local/bin/docker-clean
docker-clean

5.2 Increase VM memory when Docker pull runs out of memory

Allocate more RAM to the virtual machine before pulling the image.

5.3 Kibana cannot start (license info error)

Check that the IP address configured for Elasticsearch matches the container's IP.