Step-by-Step Guide to Build an ELK Stack with Nginx on CentOS

Environment: CentOS 6.7, IP 192.168.88.250, ElasticSearch 2.1.0, Logstash 2.1.1, Kibana 4.3.1, JDK 1.8.0_77.

Install JDK to /usr/local/java and set JAVA_HOME and PATH in /etc/profile, then source it and verify with java -version.

Install ElasticSearch:

wget https://download.elasticsearch.org/elasticsearch/distribution/tar/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.1.0/elasticsearch-2.1.0.tar.gz
tar xf elasticsearch-2.1.0.tar.gz
cd /usr/local/elasticsearch-2.1.0/
./plugin -install mobz/elasticsearch-head
./elasticsearch -Des.insecure.allow.root=true
curl -X GET 192.168.88.250:9200

Install Nginx, configure /usr/local/nginx/conf/nginx.conf with a JSON log format and proxy settings for Kibana, and set up a server block for kibana_server:

# example snippet of nginx.conf (relevant parts)
worker_processes 1;
events { worker_connections 1024; }
http {
    log_format json '{"@timestamp":"$time_iso8601","host":"$server_addr","clientip":"$remote_addr","size":$body_bytes_sent,"responsetime":$request_time,"upstreamtime":"$upstream_response_time","upstreamhost":"$upstream_addr","http_host":"$host","url":"$uri","xff":"$http_x_forwarded_for","referer":"$http_referer","agent":"$http_user_agent","status":"$status"}';
    access_log /var/log/nginx/access.log_json json;
    server {
        listen 80;
        server_name kibana_server;
        location / {
            root /var/www/kibana;
            index index.html index.htm;
        }
        location ~ ^/kibana4/.* {
            proxy_pass http://kibana4;
            rewrite ^/kibana4/(.*) /$1 break;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host;
            auth_basic "Restricted";
            auth_basic_user_file /etc/nginx/conf.d/kibana.myhost.org.htpasswd;
        }
    }
}

Install Logstash, create stdin.conf to read Nginx JSON logs, split and convert upstreamtime, and output to ElasticSearch:

wget https://download.elastic.co/logstash/logstash/logstash-2.1.1.tar.gz
tar xf logstash-2.1.1.tar.gz
cd /usr/local/logstash-2.1.1/bin
vim stdin.conf   # configure input, filter, output
# input { file { path => "/var/log/nginx/access.log_json" codec => "json" } }
# filter { mutate { split => ["upstreamtime",","] } mutate { convert => ["upstreamtime","float"] } }
# output { elasticsearch { hosts => ["192.168.88.250:9200"] index => "logstash-%{type}-%{+YYYY.MM.dd}" document_type => "%{type}" } }
./logstash -f stdin.conf &

Install Kibana, edit config/kibana.yml to point to the ElasticSearch URL, and start Kibana (listening on port 5601):

wget https://download.elastic.co/kibana/kibana/kibana-4.3.1-linux-x64.tar.gz
tar xf kibana-4.3.1-linux-x64.tar.gz
cd /usr/local/kibana-4.3.1-linux-x64/
vim config/kibana.yml   # set elasticsearch.url to http://192.168.88.250:9200
./kibana &

After all components are running, access Kibana at http://192.168.88.250:5601, use Discover to view logs, adjust the time zone (add 8 hours) if data does not appear, and customize visualizations as needed.