Step-by-Step Guide to Deploying an ELK Stack (Elasticsearch, Logstash, Kibana) with Filebeat on Multiple Linux Servers

The deployment environment consists of three servers (192.168.236.179, .180, .181) each intended to run Elasticsearch, Logstash, Kibana, Filebeat, and JDK.

JDK installation and system tuning :

tar xf jdk-1.8.0_linux-x64_bin.tar.gz -C /usr/local/

vim /etc/profile.d/jdk.sh

(add environment variables)

export JAVA_HOME=/usr/local/jdk-1.8.0/

export PATH=$PATH:$JAVA_HOME/bin

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

source /etc/profile.d/jdk.sh

Verify with java -version.

Elasticsearch installation on the first node (es1) :

tar xf elasticsearch-7.6.2-linux-x86_64.tar.gz -C /usr/local/

mv elasticsearch-7.6.2 elasticsearch

Edit /usr/local/elasticsearch/config/elasticsearch.yml to set:

cluster.name: my-elk

node.name: node-1

path.data: /data/es/data

path.logs: /data/es/log

bootstrap.memory_lock: true

network.host: 192.168.236.179

http.port: 9200

discovery.seed_hosts: ["192.168.236.179", "192.168.236.180", "192.168.236.181"]

cluster.initial_master_nodes: ["192.168.236.179", "192.168.236.180", "192.168.236.181"]

discovery.zen.minimum_master_nodes: 2

gateway.recover_after_nodes: 2

.

Create data and log directories and a dedicated user:

mkdir -pv /data/es/data

mkdir -pv /data/es/logs

useradd elastic

chown -R elastic:elastic /data/es/

chown -R elastic:elastic /usr/local/elasticsearch/

.

Start Elasticsearch service as the elastic user:

su - elastic

cd /usr/local/elasticsearch

nohup ./bin/elasticsearch > /tmp/elastic.log &

Check with netstat -nltp | grep -E "9200|9300" and curl http://192.168.236.179:9200/, which returns a JSON block showing node name, cluster name, version, etc.

Copy the Elasticsearch directory to the other two nodes:

scp -r elasticsearch [email protected]:/usr/local/

scp -r elasticsearch [email protected]:/usr/local/

.

Kibana installation :

tar xf kibana-7.6.2-linux-x86_64.tar.gz -C /usr/local/

ln -sv kibana-7.6.2-linux-x86_64/ kibana

Edit /usr/local/kibana/config/kibana.yml to set:

server.port: 5601

server.host: "172.16.150.159"

elasticsearch.hosts: ["http://172.16.150.157:9200"]

i18n.locale: "zh-CN"

. Start Kibana: nohup ./kibana --allow-root > /tmp/kibana.log &.

Filebeat installation :

tar xf filebeat-7.6.2-linux-x86_64.tar.gz -C /usr/local/

cd /usr/local/filebeat

Edit filebeat.yml as needed, then start:

nohup /usr/local/filebeat/filebeat -e -c /usr/local/filebeat/filebeat.yml > /tmp/filebeat.log &

.

Install Nginx and configure its access logs to output JSON format (details omitted).

Logstash installation :

tar xf logstash-7.6.2.tar.gz -C /usr/local/

cd /usr/local/logstash/config/

Create logstash_nginx.conf with the following sections:

input { file { path => "/opt/nginx/logs/access.log" type => "nginxlog" start_position => "beginning" } }

filter { if [type]=="nginxlog" { grok { match => ["message", "%{COMMONAPACHELOG}"] } date { match => ["timestamp","dd/MMM/yyyy:HH:mm:ss Z"] } } }

output { stdout { codec => rubydebug } elasticsearch { hosts => "192.168.236.179:9200","192.168.236.180:9200","192.168.236.181:9200" } }

Start Logstash:

nohup /usr/local/logstash/bin/logstash -f logstash_nginx.conf > /tmp/logstch.log &

.

After all components are running, access the Kibana UI to verify that logs are being ingested and visualized. The article notes that this setup can be extended to a distributed logging system using ELK + Filebeat + Kafka.