Deploying ELK Stack with Docker: Offline Installation, Configuration, and Log Collection

Overview

This article explains how to containerize and deploy an ELK Stack (Elasticsearch, Logstash, Kibana, Filebeat) using Docker, including offline installation, IK analyzer support, multiline plugin setup, and end‑to‑end log collection.

Features

Offline environment deployment

IK analyzer installation

Offline Logstash multiline plugin installation

Production‑grade Logstash and Filebeat configurations

Filebeat resynchronization

Step‑by‑step illustrated tutorial

Environment

Ubuntu 18.04 VM, Elasticsearch/Logstash/Kibana/Filebeat version 7.6.2. Five containers will be started.

1. Deploy Elasticsearch with Docker

1.1 Pull Image

docker pull elasticsearch:7.6.2

1.2 Offline Load

# From an online machine
sudo docker pull elasticsearch:7.6.2
sudo docker save -o es.tar es
sudo chmod 777 es.tar
# On the offline machine
sudo docker load -i es.tar

1.3 Create Directories and Permissions

mkdir -p /data/elk/es/{config,data,logs,plugins}
sudo chown -R 1000:1000 /data/elk/es

1.4 Configure Elasticsearch

cd /data/elk/es/config
sudo touch elasticsearch.yml
# elasticsearch.yml content
cluster.name: "my-es"
network.host: 0.0.0.0
http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"

1.5 Run Container

docker run -it -d \
  -p 9200:9200 \
  -p 9300:9300 \
  --name es01 \
  -e ES_JAVA_OPTS="-Xms1g -Xmx1g" \
  -e "discovery.type=single-node" \
  --restart=always \
  -v /data/elk/es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
  -v /data/elk/es/data:/usr/share/elasticsearch/data \
  -v /data/elk/es/logs:/usr/share/elasticsearch/logs \
  -v /data/elk/es/plugins:/usr/share/elasticsearch/plugins \
  elasticsearch:7.6.2

1.6 Verify

curl http://localhost:9200

1.7 Install ES‑Head (optional UI)

sudo docker pull mobz/elasticsearch-head:5
docker run -d -p 9100:9100 mobz/elasticsearch-head:5

2. Deploy Kibana

2.1 Pull Image

docker pull kibana:7.7.1

2.2 Create Configuration

mkdir -p /data/elk/kibana
vim /data/elk/kibana/kibana.yml
# kibana.yml content
server.name: kibana
server.host: "0"
elasticsearch.hosts: ["http://172.17.0.2:9200"]
xpack.monitoring.ui.container.elasticsearch.enabled: true

2.3 Run Container

docker run -it -d \
  --restart=always \
  --name kibana \
  -p 5601:5601 \
  -v /data/elk/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml \
  kibana:7.6.2

3. Deploy Logstash

3.1 Pull Image (online or offline)

# Online
sudo docker pull logstash:7.6.2
# Offline
sudo docker save -o logstash.tar logstash
sudo chmod 777 logstash.tar
sudo docker load -i logstash.tar

3.2 Extract Files for Configuration

docker run -d --name=logstash logstash:7.6.2
sudo docker cp logstash:/usr/share/logstash /data/elk
sudo chmod -R 777 /data/elk/logstash

3.3 Create logstash.conf

sudo mkdir -p /data/elk/logstash/conf.d
cd /data/elk/logstash/conf.d
sudo vim logstash.conf
# Sample input, filter, output (truncated for brevity)

3.4 Create logstash.yml

cd /data/elk/logstash/config
sudo vim logstash.yml
# path.config: /usr/share/logstash/conf.d/*.conf
# path.logs: /var/log/logstash
# http.host: "0.0.0.0"
# xpack.monitoring.elasticsearch.hosts: ["http://192.168.56.11:9200"]

3.5 Run Logstash Container

docker rm -f logstash
docker run -it -d \
  -p 5044:5044 \
  --name logstash \
  --restart=always \
  -v /data/elk/logstash:/usr/share/logstash \
  logstash:7.6.2

3.6 Install Multiline Plugin (online & offline)

# Online
bin/logstash-plugin install logstash-filter-multiline
# Offline (prepare on a machine with internet)
bin/logstash-plugin install logstash-filter-multiline
bin/logstash-plugin prepare-offline-pack logstash-filter-multiline
# Copy the zip to the offline server and install inside the container
docker exec -it logstash /bin/bash
cd /usr/share/logstash
bin/logstash-plugin install file:///usr/share/logstash/multiline/logstash-offline-plugins-7.6.2.zip

4. Deploy Filebeat

4.1 Pull Image

docker pull elastic/filebeat:7.6.2

4.2 Extract Files

docker run -d --name=filebeat elastic/filebeat:7.6.2
docker cp filebeat:/usr/share/filebeat /data/elk

4.3 Create filebeat.yml

vim /data/elk/filebeat/filebeat.yml
# filebeat.yml content (paths, fields, multiline, output.logstash)

4.4 Set Permissions and Run

sudo chmod -R 777 /data/elk/filebeat
sudo chmod go-w /data/elk/filebeat/filebeat.yml
docker rm -f filebeat
docker run -d \
  --name=filebeat \
  --restart=always \
  -v /data/passjava/logs:/usr/share/filebeat/passjava/logs/ \
  -v /data/elk/filebeat:/usr/share/filebeat \
  elastic/filebeat:7.6.2

5. Test Log Flow

5.1 Create Sample Log

sudo vim /data/passjava/logs/wukong/error.log

5.2 Verify Logstash Receives Logs

docker logs logstash

5.3 Verify Filebeat Output

docker logs filebeat

5.4 Create Index in Kibana

Open Kibana UI, create the passjava_log index pattern, and explore logs.

6. Troubleshooting

Intermittent Filebeat errors such as Failed to publish events caused by: EOF may appear but often recover automatically; monitor container logs for details.