Understanding Logstash: Core Syntax, Filters, and Advanced Configuration

Logstash is a powerful open‑source data processing pipeline that efficiently parses, filters, and transports log data, widely used in the ELK (Elasticsearch, Logstash, Kibana) stack.

1. Logstash Configuration Basics – A Logstash configuration file consists of three sections: input , filter , and output . The input defines the log source, filter processes the data, and output determines where the data is sent.

Example configuration:

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{SYSLOGLINE}" }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "syslog-%{+YYYY.MM.dd}"
  }
}

This example reads from /var/log/syslog , uses the grok filter to parse syslog format, and outputs the result to Elasticsearch.

2. Filters: The Core of Complex Log Processing

2.1 Grok – Extract Structured Data Grok is the most common filter, using regex‑like patterns to extract fields from unstructured logs.

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}

The built‑in COMBINEDAPACHELOG pattern parses Apache access logs. Logstash provides many predefined grok patterns for common log formats.

2.2 Mutate – Data Transformation and Enrichment The mutate filter can rename fields, remove unwanted data, convert types, etc.

filter {
  mutate {
    rename => { "clientip" => "client_ip" }
    convert => { "response_bytes" => "integer" }
    remove_field => ["unnecessary_field"]
  }
}

These operations adjust the data to suit downstream analysis.

2.3 Date – Timestamp Handling The date filter parses various timestamp formats into the standard @timestamp field.

filter {
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    target => "@timestamp"
  }
}

This converts the timestamp field to a unified time format for consistent analysis.

2.4 Ruby – Custom Logic When built‑in filters are insufficient, Ruby code can be embedded for flexible processing.

filter {
  ruby {
    code => "event.set('combined_field', event.get('field1') + ' ' + event.get('field2'))"
  }
}

The example creates a new field by concatenating two existing fields.

2.5 Aggregate – Multi‑Event Aggregation The aggregate filter groups related events, useful for multi‑line logs.

filter {
  aggregate {
    task_id => "%{some_id}"
    code => "map['count'] ||= 0; map['count'] += 1"
    push_map_as_event_on_timeout => true
    timeout => 120
  }
}

Events with the same task_id are combined within 120 seconds, producing a single event with a count.

3. Conditional Statements and Multi‑Event Handling Logstash supports conditional logic to apply different filters or outputs based on event content.

filter {
  if [source] == "apache_logs" {
    grok { match => { "message" => "%{COMMONAPACHELOG}" } }
  } else if [source] == "syslog" {
    grok { match => { "message" => "%{SYSLOGLINE}" } }
  }
}

Conditional statements enable flexible routing, and Logstash also supports multiple pipelines to connect several instances for complex event flows.

Conclusion By mastering these syntaxes and filters, you can tackle a wide range of log‑processing challenges, making data analysis smoother and more effective.