Step-by-Step Guide to Deploying the ELK Stack (Elasticsearch, Logstash, Kibana, Filebeat) on Ubuntu

The author introduces the ELK (Elasticsearch‑Logstash‑Kibana) log platform, explaining that it offers a complete solution for log collection, processing, and visualization, and notes that the tutorial will present a step‑by‑step, picture‑rich implementation on a single Ubuntu virtual machine with 6 GB memory.

ELK Stack Overview

The ELK Stack consists of Elasticsearch for storage and search, Logstash for data ingestion and transformation, Filebeat as a lightweight log shipper, and Kibana for visual dashboards. An alternative architecture using Kafka is mentioned but not covered in this article.

The data flow is: Filebeat → Logstash (filter/transform) → Elasticsearch (store) → Kibana (visualize).

1. Deploy Elasticsearch

Pull the Elasticsearch Docker image and create a mounted directory structure:

docker pull elasticsearch:7.7.1

mkdir -p /data/elk/es/{config,data,logs}

chown -R 1000:1000 /data/elk/es

Create elasticsearch.yml with basic settings:

cluster.name: "my-es"
network.host: 0.0.0.0
http.port: 9200

Run the container:

docker run -it -d -p 9200:9200 -p 9300:9300 --name es \
  -e ES_JAVA_OPTS="-Xms1g -Xmx1g" \
  -e "discovery.type=single-node" \
  --restart=always \
  -v /data/elk/es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
  -v /data/elk/es/data:/usr/share/elasticsearch/data \
  -v /data/elk/es/logs:/usr/share/elasticsearch/logs \
  elasticsearch:7.7.1

Verify with curl http://localhost:9200.

2. Deploy Kibana

Pull the Kibana image: docker pull kibana:7.7.1 Obtain the Elasticsearch container IP (e.g., 172.17.0.2) using:

docker inspect --format '{{ .NetworkSettings.IPAddress }}' es

Create kibana.yml:

#Default Kibana configuration for docker target
server.name: kibana
server.host: "0"
elasticsearch.hosts: ["http://172.17.0.2:9200"]
xpack.monitoring.ui.container.elasticsearch.enabled: true

Run Kibana:

docker run -d --restart=always \
  --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 \
  --name kibana -p 5601:5601 \
  -v /data/elk/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml \
  kibana:7.7.1

Access http://<your‑host-ip>:5601, import the sample data, and explore the Discover view.

3. Deploy Logstash

Install Java JDK 8, then download and extract Logstash 7.7.1:

sudo apt install openjdk-8-jdk

curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-7.7.1.tar.gz

tar -xzvf logstash-7.7.1.tar.gz

Test a simple pipeline:

cd logstash-7.7.1
bin/logstash -e 'input { stdin { } } output { stdout {} }'

Create weblog.conf (input via TCP 9900, grok filter, mutate, geoip, useragent, date, and outputs to stdout and Elasticsearch). Example snippet:

input {
  tcp { port => 9900 }
}
filter {
  grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
  mutate { convert => { "bytes" => "integer" } }
  geoip { source => "clientip" }
  useragent { source => "agent" target => "useragent" }
  date { match => ["timestamp","dd/MMM/yyyy:HH:mm:ss Z"] }
}
output {
  stdout { }
  elasticsearch { hosts => ["localhost:9200"] }
}

Run Logstash with the configuration: bin/logstash -f weblog.conf Send a test line using netcat: head -n 1 weblog-sample.log | nc localhost 9900 Verify the document appears in Elasticsearch via GET logstash/_search.

4. Deploy Filebeat

Download and extract Filebeat 7.7.1:

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.1-linux-x86_64.tar.gz
tar xzvf filebeat-7.7.1-linux-x86_64.tar.gz

Basic configuration to ship logs directly to Elasticsearch:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/vagrant/logs/*.log

output.elasticsearch:
  hosts: ["192.168.56.10:9200"]

Start Logstash first, then run Filebeat: ./filebeat -e -c filebeat_apache.yml Check indices with curl http://localhost:9200/_cat/indices?v and create an index pattern in Kibana to view the logs.

5. Common Issues & Solutions

Failed Kibana image pull due to out‑of‑memory: clean up inodes or increase VM memory.

License errors: ensure Kibana points to the correct Elasticsearch IP.

Reference links to detailed troubleshooting are provided.

Future Enhancements

Add Kafka for high‑throughput ingestion.

Integrate Grafana for additional monitoring.

Implement distributed tracing.

After completing the steps, the ELK Stack is fully operational, allowing efficient log search and visualization.