Step‑by‑Step Guide to Building a Filebeat‑Kafka‑ELK Logging Pipeline

Recent surveys show Filebeat has replaced Logstash‑Forwarder as the modern log shipper, prompting many to adopt a Filebeat‑Kafka‑ELK architecture. The stack consists of Elasticsearch (core search engine), Logstash (log processing), Kibana (visualization), Filebeat (lightweight collector), and Kafka (high‑throughput message broker).

Version Compatibility

Use the following versions to avoid incompatibilities: Filebeat 7.8.1, Logstash 7.8.1, Kibana 7.8.1, Elasticsearch 7.8.1 (requires Java 11), and Kafka 2.2.2 (compatible with Filebeat).

Filebeat Installation & Configuration

1. Extract the Filebeat package.

2. Edit filebeat.yml to define the log source and Kafka output:

# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /yw/*.log

output.kafka:
  enabled: true
  hosts: ["localhost:9092"]
  topic: "test"

3. Start Filebeat with the configuration file:

nohup ./filebeat -c filebeat.yml &

Kafka Installation & Configuration

1. Extract Kafka.

2. Modify config/server.properties to set the listener port: listeners=PLAINTEXT://localhost:9092 3. If Zookeeper is not installed, use Kafka’s bundled Zookeeper and edit config/zookeeper.properties:

# Directory for snapshots
 dataDir=/tmp/zookeeper
 clientPort=2181
 maxClientCnxns=0

4. Start Zookeeper and Kafka:

cd bin
nohup ./zookeeper-server-start.sh ../config/zookeeper.properties &
nohup ./kafka-server-start.sh ../config/server.properties &

5. Verify the topic exists:

./kafka-topics.sh --list --bootstrap-server localhost:9092
./kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic test

Logstash Installation & Configuration

1. Extract Logstash.

2. Edit config/logstash.yml to set the node name.

3. Create a pipeline configuration test.conf that reads from Kafka and writes to Elasticsearch:

input {
  kafka {
    bootstrap_servers => ["localhost:9092"]
    group_id => "test"
    topics => ["test"]
    consumer_threads => 1
    codec => json { charset => "UTF-8" }
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "test-%{+YYYY.MM}"
  }
}

4. Start Logstash with the custom pipeline:

cd ../bin
nohup ./logstash -f ../config/test.conf &

Elasticsearch Installation & Configuration

1. Extract Elasticsearch.

2. Create a dedicated user and assign ownership:

useradd elasticsearch
chown -R elasticsearch elasticsearch/

3. Switch to the elasticsearch user and edit config/elasticsearch.yml to set the HTTP port (9200).

4. Limit JVM heap for a low‑resource environment:

-Xms256m
-Xmx512m

5. Start Elasticsearch and verify with curl:

./elasticsearch
curl localhost:9200

Kibana Installation & Configuration

1. Extract Kibana.

2. Create a kibana user and set ownership.

3. Edit config/kibana.yml to point to Elasticsearch: elasticsearch.hosts: ["http://localhost:9200"] 4. Start Kibana:

cd bin
./kibana

5. Open a browser at http://localhost:5601 to access the UI.

Validation of the End‑to‑End Flow

Generate some log entries, then consume messages from Kafka to confirm they were forwarded:

./kafka-console-consumer.sh --topic test --bootstrap-server localhost:9092

In Kibana Discover, select the index test-YYYY.MM, filter on message containing "INFO", and locate the entry with "37 milliseconds" to verify successful ingestion and visualization.

All components are now running, and logs can be searched and analyzed through Kibana.