How to Monitor Nginx Logs with ELK: From Logstash Config to Kibana Dashboard

1. Introduction

This article introduces how to monitor Nginx logs, analyze them with Logstash, and display the results in visual graphs. It also shows how to use Nginx as a reverse proxy for Kibana with HTTP basic authentication.

Note: The environment assumes the ELK stack (Elasticsearch, Logstash, Kibana) is already installed, along with the required Java JDK.

2. Configure Logstash

1. Create a new configuration file under /etc/logstash/conf.d :

[root@log-monitor ~]# cat /etc/logstash/conf.d/nginx_access.conf
input {
    file {
        path => [ "/data/nginx-logs/access.log" ]
        start_position => "beginning"
        ignore_older => 0
    }
}

filter {
    grok {
        match => { "message" => "%{NGINXACCESS}" }
    }
    geoip {
      source => "http_x_forwarded_for"
      target => "geoip"
      database => "/etc/logstash/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
      convert => [ "response", "integer" ]
      convert => [ "bytes", "integer" ]
      replace => { "type" => "nginx_access" }
      remove_field => "message"
    }
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    }
    mutate { remove_field => "timestamp" }
}
output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "logstash-nginx-access-%{+YYYY.MM.dd}"
    }
    stdout { codec => rubydebug }
}

2. Explanation of the configuration

Input section

path

: Path to the log file, e.g., /data/nginx-logs/access.log. start_position: Read from the beginning of the file. ignore_older: Set to 0 to disable ignoring older files.

Filter section

grok

: Parses the Nginx log using the NGINXACCESS pattern. geoip: Enriches the log with geographic information based on the client IP. source: Field containing the IP address (usually the last field in the log). target: Stores the GeoIP data under the geoip field. database: Path to the GeoIP database. add_field: Adds longitude and latitude as separate fields. mutate: Converts field types, replaces the type field, and removes the original message field. date: Parses the timestamp field into a proper date format.

Output section

elasticsearch

: Sends the processed data to Elasticsearch, creating an index named logstash-nginx-access-YYYY.MM.DD. stdout: Prints events to the console for debugging.

3. Create the Grok pattern

After creating the Logstash config, add a Grok pattern file:

[root@log-monitor ~]# mkdir -pv /opt/logstash/patterns
[root@log-monitor ~]# cat /opt/logstash/patterns/nginx
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} - %{NOTSPACE:remote_user} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} "%{IPV4:http_x_forwarded_for}"

Note: The pattern includes http_x_forwarded_for because the logs are behind a CDN; the last field contains the real client IP.

3. Configure Elasticsearch

1. Modify elasticsearch.yml

[root@log-monitor ~]# egrep -v '^#|^$' /etc/elasticsearch/elasticsearch.yml
node.name: es-1
path.data: /data/elasticsearch/
network.host: 127.0.0.1
http.port: 9200

Create the data directory and set permissions:

[root@log-monitor ~]# mkdir -pv /data/elasticsearch
[root@log-monitor ~]# chown -R elasticsearch.elasticsearch /data/elasticsearch/

Restart services and verify they are listening:

[root@log-monitor ~]# systemctl restart elasticsearch
[root@log-monitor ~]# systemctl restart logstash
[root@log-monitor ~]# netstat -ulntp | grep java
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 25988/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 25988/java

4. Install Nginx and Configure Kibana Proxy

1. Install Nginx

[root@log-monitor ~]# wget https://nginx.org/packages/rhel/7/x86_64/RPMS/nginx-1.10.0-1.el7.ngx.x86_64.rpm
[root@log-monitor ~]# yum localinstall nginx-1.10.0-1.el7.ngx.x86_64.rpm -y

Create elk.conf to proxy Kibana:

[root@log-monitor ~]# cat /etc/nginx/conf.d/elk.conf
upstream elk {
    ip_hash;
    server 172.17.0.1:5601 max_fails=3 fail_timeout=30s;
    server 172.17.0.1:5601 max_fails=3 fail_timeout=30s;
}

server {
    listen 80;
    server_name localhost;
    server_tokens off;
    client_body_timeout 5s;
    client_header_timeout 5s;
    location / {
        proxy_pass http://elk/;
        index index.html index.htm;
        auth_basic "ELK Private,Don't try GJ!";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }
}

2. Set up HTTP basic authentication

[root@log-monitor ~]# yum install httpd-tools -y
[root@log-monitor ~]# htpasswd -cm /etc/nginx/.htpasswd elk
New password:
Re-type new password:
Adding password for user elk

Start Nginx and verify the proxy port (e.g., 8888):

[root@log-monitor ~]# systemctl start nginx
[root@log-monitor ~]# netstat -ultpn | grep :8888
tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 26424/nginx: master

3. Configure firewall

[root@log-monitor ~]# iptables -I INPUT -p tcp -m state --state NEW --dport 8888 -j ACCEPT

5. Verify access

Open a browser, log in with the elk user, and you should see the Kibana interface. Add the index pattern logstash-nginx-access-* and set it as the default. The Discover view will display the ingested Nginx logs.

Sample Kibana dashboard visualizations include client IP geographic distribution, total HTTP requests, top source IPs, top requested pages, and error trends.

5. Summary

Advantages of the ELK stack for operations:

Facilitates rapid forensic analysis during security incidents.

Centralizes log collection and storage for easy downstream analysis.

Provides data‑driven insights for performance tuning and system optimization.