Master ELK Stack on Ubuntu: Step-by-Step Logstash & Filebeat Setup

Installation environment: Ubuntu 16.04.2 LTS.

Preface

ELK is the combination of Elasticsearch, Logstash, and Kibana, an open‑source log management solution that can ingest logs from any source, search, analyze, and display them in real time.

Key components:

1. Filebeat – monitors and forwards log files to Logstash. 2. Logstash – collects, processes, and forwards logs to Elasticsearch. 3. Elasticsearch – provides distributed full‑text search, indexing and aggregation. 4. Kibana – web‑based UI for visualizing Elasticsearch data.

Official website: https://www.elastic.co

Documentation links for each component are provided.

ELK Architecture Relationship

The simplest ELK architecture consists of five steps: Filebeat reads Tomcat logs, forwards them to Logstash, Logstash processes and sends them to Elasticsearch, Elasticsearch stores and indexes the data, and Kibana visualizes the results.

Logstash Installation and Usage

Ensure JDK 1.8+, download Logstash from https://www.elastic.co/downloads/logstash, and extract it.

Example command to extract: tar -xvf logstash-5.1.1.tar.gz Run a basic pipeline to test the installation. The -e parameter allows inline configuration; the example reads from stdin and outputs to stdout.

After confirming the pipeline works, you can create a logstash.conf file and run Logstash with:

bin/logstash -f logstash.conf

Logstash pipelines consist of inputs, optional filters, and outputs. Input plugins consume data, filters modify it, and output plugins write it to destinations.

Using Filebeat to Send Logs to Logstash

Filebeat is a lightweight shipper that reads log files and forwards them to Logstash. Install Filebeat from https://www.elastic.co/downloads/beats/filebeat and extract it.

Edit filebeat.yml to specify the log file paths (e.g., /home/xuliugen/www/ufind/logs/*.log) and to enable the Logstash output (comment out output.elasticsearch and configure the Logstash host and port 5044).

Start Filebeat with: ./filebeat -e -c filebeat.yml -d "publish" Modify the Logstash configuration to use the beats input on port 5044 and output to stdout with codec => rubydebug for debugging.

Validate the configuration with bin/logstash -f logstash.conf --config.test_and_exit and then start Logstash with automatic config reload.

When Logstash and Filebeat are running, log entries from Tomcat appear in the console, confirming successful integration.