How to Install and Configure Filebeat with ELK Using Docker Compose

Background

In the previous article we built an ELK log collection platform; this chapter focuses on sending log files to that ELK platform using Filebeat.

Filebeat

Filebeat is a lightweight log shipper in the Beats family that forwards log files from many client machines to an ELK stack via Elasticsearch or Logstash ports. This article covers the integration of Filebeat with Logstash.

Introduction

Filebeat’s back‑pressure‑sensitive protocol slows reading when Logstash is busy, allowing Logstash to recover before the shipper resumes normal speed.

Prerequisite

Install and start Filebeat service

Install the official RPM package:

1. Use RPM to install the official package

2. Configure Filebeat to point to Logstash

Edit /etc/filebeat/filebeat.yml to comment out Elasticsearch settings and set the Logstash host (e.g., localhost). Configure the log paths, e.g., /var/log/*.log, using glob patterns.

Example Logstash host configuration:

Configure the log directory with proper indentation.

3. Start the Filebeat service

Service started successfully.

Using docker‑compose

1. Clone the Git repository to a directory such as /home/user1/.

2. Modify the environment file ~/filebeat/.env with the following key properties:

environment: distinguishes log environments, similar to ELK.

LOGSTASH_HOST: Logstash host (default localhost, port 5044).

VOLUME_FOLDER_LIST: absolute paths of folders containing logs to ship.

PROSPECT_FILE_PATTERN_LIST: specific file patterns to read, using glob syntax.

3. Run gen-config.sh to generate filebeat.yml and docker-compose.yml.

Review the generated ~/filebeat/filebeat.yml and ~/filebeat/docker-compose.yml files.

4. Start Filebeat with docker‑compose

Viewing logs in Kibana

If ELK has not yet collected logs, create an index when accessing Kibana at localhost:5601.

Then use the Discovery view to see the logs shipped by Filebeat.

At this point the Filebeat service is installed and running; choose the installation method that best fits your needs.