How to Build a Lightweight Log Monitoring Stack with Logstash, Elasticsearch, and Kibana

Log analysis and monitoring are essential in system development; the more complex the system, the more critical they become.

Keyword‑based log detail queries

System health monitoring

Statistical analysis such as call counts, execution time, success rate

Automatic alerts for abnormal data

Log‑driven data mining

Common problems teams face include:

Developers cannot log into production servers to view detailed logs, requiring time‑consuming hand‑offs with operations

Log data scattered across multiple systems, making search difficult

Large log volumes leading to slow queries

Cross‑system calls make pinpointing data across logs hard

Logs not being real‑time

Heavyweight open‑source tracing systems (e.g., Facebook Scribe, Cloudera Flume, Twitter Zipkin, Storm) are powerful but often overly complex for many teams.

For most scenarios, a lightweight, ready‑to‑use stack—Logstash + Elasticsearch + Kibana (LEK)—is recommended.

Logstash

Logstash is simple to deploy (a single JAR) and processes logs through a pipeline:

inputs >> codecs >> filters >> outputs

It supports common log types, integrates easily with monitoring tools (Zabbix, Nagios, email), and works well with Redis as an input buffer and Graphite for visualizing metrics.

Elasticsearch

Elasticsearch is a Lucene‑based open‑source search engine with features such as real‑time indexing, distribution, high availability, document orientation, schema‑free design, and a RESTful API.

Useful Chinese analysis plugins include smartcn, mmseg, ik, pinyin, and stconvert. Helpful tools: elasticsearch‑servicewrapper, Elastic HQ, and elasticsearch‑rtf.

Kibana

Kibana is a powerful client for visualizing Elasticsearch data. Kibana 3 is a pure HTML + JavaScript client that can be served from any HTTP server (Apache, Nginx, etc.). Demo URLs and screenshots illustrate charts, tables, query view, panels, pie charts, maps, and rich query syntax.

Installation & Deployment

Install JDK 1.7

java -version

sudo vim ~/.bashrc

export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

source ~/.bashrc

Install Redis

cd ~/src
wget http://download.redis.io/releases/redis-2.6.16.tar.gz
tar -zxf redis-2.6.16.tar.gz
cd redis-2.6.16
make
sudo make install

cd utils
sudo ./install_server.sh

sudo /etc/init.d/redis_ start
sudo /etc/init.d/redis_ end

Install Elasticsearch

cd /search
sudo mkdir elasticsearch
cd elasticsearch
sudo wget http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.5.zip
sudo unzip elasticsearch-0.90.5.zip
cd elasticsearch-0.90.5
bin/elasticsearch -f

curl -X GET http://localhost:9200

Install Logstash

cd /search
sudo mkdir logstash
cd logstash
sudo wget http://download.elasticsearch.org/logstash/logstash/logstash-1.2.1-flatjar.jar

agent   # run in Agent mode
-f CONFIGFILE # specify config file
web     # start built‑in web UI
-p PORT # port (default 9292)

Install Kibana

cd /search
sudo mkdir kibana
sudo wget http://download.elasticsearch.org/kibana/kibana/kibana-latest.zip
sudo unzip kibana-latest.zip
sudo cp -r kibana-latest /var/www/html
# edit config.js to point to Elasticsearch

Integration

Start Redis and Elasticsearch, then create a Logstash configuration (redis.conf):

input {
  redis {
    host => "127.0.0.1"
    port => "6379"
    key => "logstash:demo"
    data_type => "list"
    codec => "json"
    type => "logstash-redis-demo"
    tags => ["logstashdemo"]
  }
}

output {
  elasticsearch {
    host => "127.0.0.1"
  }
}

Run Logstash:

java -jar /search/logstash/logstash-1.2.1-flatjar.jar agent -f /search/logstash/redis.conf &

java -jar /search/logstash/logstash-1.2.1-flatjar.jar web &

Push a test log into Redis:

RPUSH logstash:demo '{"time": "2013-01-01T01:23:55", "message": "logstash demo message"}'

Check data in Elasticsearch and view it via Logstash web UI or Kibana.

Data Cleanup & Index Management

Logstash creates daily Elasticsearch indices, making old‑data deletion simple by dropping whole indices. Elasticsearch also supports document TTL, though bulk index deletion is more I/O intensive.

Set fields like URLs to not_analyzed to avoid tokenization, use multi‑field mappings, and prefer bulk imports for large volumes. Filters are faster than queries because they skip scoring and are cached.

Adjust replication to asynchronous mode to speed up bulk log ingestion.

Elasticsearch Optimization

Optimize the JVM, increase the maximum number of file descriptors, and tune the index refresh interval.

Best Practices

Ensure your applications emit logs.

Log messages must be useful for troubleshooting; generic "parameter error" logs are insufficient.

Do not rely solely on exceptions; log key parameters such as timestamps, execution time, source, input/output values, error codes, and stack traces.

Include identifiers like sessionId, transactionId, userId to correlate logs across services.

Prefer plain‑text or JSON formats.

Use a queue (e.g., Redis) for buffering.

Other Log‑Related Tools

rsyslog

syslog‑ng

graylog

fluentd

nxlog