How to Install and Use Logstash: From Console to Elasticsearch and Redis

Introduction

Logstash is an open‑source data collection engine.

It works like a pipe: the left side connects to data sources, the right side to storage destinations, and filters in the middle process the data, storing only what meets the criteria.

Typical sources on the left include log files , Redis , Kafka , etc.; destinations on the right include Elasticsearch , Kafka , MongoDB , and more.

The core components of Logstash are input, filter, and output.

Logstash provides over 200 plugins, so most integrations require only simple configuration; custom plugins can be developed when needed.

Installation

Logstash does not require a separate installer; with Java installed, just download and extract.

Download: https://www.elastic.co/downloads/logstash

Extract: tar zxvf logstash-5.0.2.tar.gz Run

Navigate to the extracted directory and execute the binary:

cd logstash-5.0.2
bin/logstash -e 'input { stdin { } } output { stdout {} }'

This command starts Logstash with a simple configuration that reads from standard input and writes to standard output.

After a short startup, you will see “Successfully started Logstash …”. Type a string, e.g., hello world and you will receive a timestamped output like:

2016-12-07T08:38:18.711Z MacbookPro.local hello world

Output to Elasticsearch

Replace the output with Elasticsearch:

bin/logstash -e 'input { stdin { } } output { elasticsearch { } }'

The input remains stdin; the output uses default Elasticsearch settings (localhost:9200). Use the host parameter to change the address.

After starting, Logstash creates an index prefixed with logstash. Verify with: curl 'localhost:9200/_cat/indices?v' Enter a string; it will be indexed. Query it with:

curl -XGET 'localhost:9200/logstash-2016.12.03/_search?pretty' -d'
{
  "query": { "match_all": {} },
  "size": 100
}'

The entered string appears in the search results.

Reading Data from Redis

Change the input to Redis while keeping the console output for easy viewing.

Create a configuration file conf/redis.conf:

input {
    redis {
        data_type => "pattern_channel"
        key => "logstash-*"
        host => "localhost"
        port => 32768
    }
}
output {
  stdout { codec => rubydebug }
}

The input section defines the Redis connection and key pattern; the output uses the rubydebug codec to format the output.

Run

bin/logstash -f conf/redis.conf

After Logstash starts, publish a message to Redis: PUBLISH logstash-demo "hello world" The Logstash console will display the received event:

{
    "@timestamp" => 2016-12-07T09:37:51.533Z,
    "@version" => "1",
    "message" => "hello world",
    "tags" => [
        "_jsonparsefailure"
    ]
}

Summary

We demonstrated three scenarios:

1) Simple stdin to stdout. 2) stdin to Elasticsearch. 3) Redis input to stdout.

These examples showcase Logstash’s ease of use and flexibility. The ELK stack is a powerful combination, and this article provides a foundation for further exploration of Logstash’s capabilities.