Mastering Log Management: Step-by-Step ELK Stack Installation on CentOS

Big Data Characteristics

Perhaps we have been confused by the hype around big data and visualization. First, what exactly is big data and how large must a dataset be to be considered big data? Below are several key characteristics.

Massive data volume (TB, PB, even EB) that requires analysis and processing.

Fast response requirements; market changes demand rapid analysis, putting performance pressure on data processing.

Data variety: multiple sources need cleaning, organization, and transformation into structured data.

Low value density: incomplete, untimely, or discontinuous samples may be noisy, but large scale can yield more reliable feedback.

Consider the following real‑world scenario.

“I am a ops engineer in a small third‑tier city, working for a startup with 10+ servers. Each server has a different configuration, and deployments change constantly. Without automation, troubleshooting involves manually checking logs on each machine, leading to endless overtime and no compensation.”

The primary goals are log centralization and rapid fault localization through search, filtering, and statistical analysis.

ELK Stack Overview

ELK refers to three open‑source tools: Elasticsearch, Logstash, and Kibana.

Elasticsearch

Open‑source distributed search engine, features:

Distributed

Zero‑configuration

Auto‑discovery

Automatic index sharding

Replica mechanism

RESTful API

Multiple data sources

Automatic load balancing

Logstash

Fully open‑source, used for log collection, analysis, and storage.

Collect

Analyze

Store

Kibana

Web UI that visualizes logs stored in Elasticsearch.

All three components are open‑source under the Apache license and provide an out‑of‑the‑box solution for centralized log collection and search, ideal for startups and small teams.

Architecture Diagram

Explanation:

Multiple independent agents (Shipper) run on each server to collect logs from various sources.

A central agent (Indexer) aggregates data; a Broker (Redis) buffers between agents.

Elasticsearch stores and indexes the data; Kibana provides a rich UI for querying and visualizing.

Installation Steps

Prerequisites: CentOS 6.6 x64 minimal installation, network and user configuration.

# yum -y update
# mkdir tmp
# cd tmp
# yum -y install gcc gcc-c++ vim tpl kernel-devel glibc-static libstdc++-static screen wget bzip2*
# screen -S elk
# wget http://down.moonstack.org/packages/jdk8u65.tar.gz
# wget http://mirrors-usa.go-parts.com/gcc/releases/gcc-4.9.3/gcc-4.9.3.tar.gz
# wget http://download.redis.io/releases/redis-3.0.6.tar.gz
# wget https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tgz
# wget https://nodejs.org/dist/v4.2.4/node-v4.2.4-linux-x64.tar.gz
# wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.4.noarch.rpm
# wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.6-1.noarch.rpm
# wget https://download.elastic.co/kibana/kibana/kibana-4.1.4-linux-x64.tar.gz

Install JDK 8u65

# tar zxvf jdk8u65.tar.gz
# mv jdk1.8.0_65/ /usr/local/java
# vim /etc/profile   (add JAVA_HOME, CLASSPATH, PATH)
# source /etc/profile
# java -version

Compile and Install GCC 4.9.3

# tar zxvf gcc-4.9.3.tar.gz
# mkdir gccmaker
# cd gcc-4.9.3/
# ./contrib/download_prerequisites
# cd ../gccmaker/
# ../gcc-4.9.3/configure --enable-checking=release --enable-languages=c,c++ --disable-multilib && make && make install
# rm -rf /usr/lib64/libstdc
# cp ./x86_64-unknown-linux-gnu/libstdc++-v3/src/.libs/libstdc++.so /usr/lib64/libstdc++.so
# ln -sv /usr/lib64/libstdc++.so /usr/lib64/libstdc++.so.6
# gcc -v

Compile and Install Python 2.7.11

# tar zxvf Python-2.7.11.tgz
# cd Python-2.7.11
# ./configure && make && make install
# python --version
# vi /usr/bin/yum   (change shebang to #!/usr/bin/python2.6)

Install Node.js 4.2.4

# tar zxvf node-v4.2.4-linux-x64.tar.gz
# mv node-v4.2.4-linux-x64/ /usr/local/nodejs
# ln -sv /usr/local/nodejs/bin/node /usr/bin/node
# ln -sv /usr/local/nodejs/bin/npm /usr/bin/npm

Compile and Install Redis 3.0.6

# tar zxvf redis-3.0.6.tar.gz
# cd redis-3.0.6
# make PREFIX=/usr/local/redis install
# ln -sv /usr/local/redis/bin/redis-server /usr/bin/redis-server
# ln -sv /usr/local/redis/bin/redis-cli /usr/bin/redis-cli

Install ELK Components

# rpm -ivh elasticsearch-1.7.4.noarch.rpm
# rpm -ivh logstash-1.5.6-1.noarch.rpm
# tar zxvf kibana-4.1.4-linux-x64.tar.gz
# mv kibana-4.1.4-linux-x64/ /usr/local/kibana
# ln -sv /usr/local/kibana/bin/kibana /usr/bin/kibana

All components are now installed; the next article will cover configuration and usage.