Configuring Nginx JSON Logging with Filebeat and Logstash for ELK Stack

When managing servers with Nginx, logs are essential for statistics, auditing, and troubleshooting; converting logs to JSON format improves visibility and is a standard practice for log collection.

Filebeat maintains the state of each file, periodically flushing it to a registry file so it can resume reading from the last offset after restarts, ensuring no log lines are lost even if the output (e.g., Elasticsearch or Logstash) is temporarily unavailable.

log_format json '{"@timestamp":"$time_iso8601","clientip":"$remote_addr","status":$status,"bodysize":$body_bytes_sent,"referer":"$http_referer","ua":"$http_user_agent","handletime":$request_time,"url":"$uri"}';

access_log  /usr/local/nginxlogs/access.log  json;

Filebeat can be configured to harvest the JSON logs produced by Nginx:

filebeat.inputs:
- type: log
  tail_files: true
  backoff: "2s"
  paths:
    - /usr/local/nginx/logs/access.log
output.logstash:
  hosts: ["192.168.20.179:5044"]

Logstash then parses the incoming JSON messages and forwards them to Elasticsearch:

input {
  beats {
    host => '0.0.0.0'
    port => 5044
  }
}
filter {
  json {
    source => "message"
    remove_field => ["message","@version","path","beat","input","log","offset","prospector","source","tags"]
  }
}
output {
  elasticsearch {
    hosts => ["http://192.168.20.182:9200","http://192.168.20.181:9200","http://192.168.20.180:9200"]
  }
}

This pipeline enables structured, searchable logs in Elasticsearch, facilitating efficient monitoring and debugging of Nginx services.