Step-by-Step Guide to Deploying ELK Stack, Logstash, Kafka, and Filebeat on CentOS

This document provides a complete deployment guide for the ELK stack (Elasticsearch, Logstash, Kibana) together with Kafka and Filebeat on a CentOS 7.2 system.

Environment Preparation

elasticsearch-6.0.0.tar.gz

filebeat-7.0.1-linux-x86_64.tar.gz

kibana-6.0.0-linux-x86_64.tar.gz

logstash-6.0.0.tar.gz

kafka_2.11-2.1.1.tgz

All packages except Kafka can be downloaded from the official Elastic website; Kafka is obtained from Apache. The guide assumes a 64‑bit CentOS 7 system; verify the OS version with cat /etc/centos-release and the architecture with getconf LONG_BIT.

Installation and Deployment

Because the stack is Java‑based, ensure JDK 1.8 is installed. Place all extracted files under /usr/elk/.

Kibana Deployment

Extract the archive and modify /conf/kibana.yml to set the Elasticsearch address and server host:

# tar -zxvf kibana-6.0.0-linux-x86_64.tar.gz
# kibana.yml edits
elasticsearch.url: "http://localhost:9200"
server.host: 0.0.0.0

Elasticsearch Deployment

Extract the archive, create a non‑root user, assign ownership, and start Elasticsearch as that user:

# tar -zxvf elasticsearch-6.0.0.tar.gz
# groupadd elkgroup
# useradd elkuser -g elkgroup -p 123456
# chown -R elkuser:elkgroup elasticsearch-6.0.0
# su elkuser
[elkuser@localhost elasticsearch-6.0.0]$ ./bin/elasticsearch

Logstash Deployment

Extract the archive, create logstash.conf under the conf directory, and start Logstash with the configuration file:

input{
  file{
    type=>"log"
    path=>"/usr/logs/*.log"
    start_position=>"beginning"
  }
}
output{
  stdout{codec=>rubydebug{}}
  elasticsearch{hosts=>"127.0.0.1" index=>"log-%{+YYYY.MM.dd}"}
}

# ./logstash -f ../config/logstash.conf

Kafka Deployment

Install the 32‑bit glibc compatibility library, configure Zookeeper, then configure and start Kafka. Adjust the heap size to avoid out‑of‑memory errors:

# yum -y install glibc.i686
# vi config/zookeeper.properties   (set dataDir, clientPort, etc.)
# ./bin/zookeeper-server-start.sh config/zookeeper.properties
# vi config/server.properties   (set broker.id, ports, log dirs, zookeeper.connect, etc.)
# export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M"
# ./bin/kafka-server-start.sh config/server.properties

Filebeat Deployment

Edit filebeat.yml to enable Kafka output, specify the log path and tags, then start Filebeat:

# vi filebeat.yml
output.kafka:
  enabled: true
  hosts: ["127.0.0.1:9092"]
  topic: test
enabled: true
paths:
  - /home/elk/log/access.log
tags: ["nginx-accesslog"]
# ./filebeat -e

Issues and Tips

When starting Logstash, a connection error may appear if the Kafka host configured in Logstash (127.0.0.1) does not match the Kafka broker’s advertised host (e.g., 10.12.1.52). Ensure both configurations use the same address and that the Kafka topic names match.

Check Logstash configuration for correct Kafka address.

Make sure Kafka’s queue (topic) name aligns with Logstash output.