Master Logstash Grok: Step‑by‑Step Guide to Parsing Logs Efficiently

Logstash Filter Plugin Grok

Logstash provides a series of filter plugins to process collected log events. By splitting events into fields, Kibana can visualize and analyze the data. This article focuses on the Grok filter.

Grok Basic Introduction

1. Grok uses text‑fragment patterns to parse log events. The syntax consists of a SYNTAX part that defines the type of the matched value (e.g., NUMBER, IP) and a SEMANTIC part that declares a variable name to store the value in Elasticsearch, making it searchable in Kibana.

For example, an event log record containing an IP address can be parsed with the pattern %{IP:client_ip_address}, storing the value in the client_ip_address field.

A matching Grok pattern for such a record might look like the following image:

Place the filter configuration in a .conf file under the Logstash conf.d directory:

The resulting parsed fields are shown in the filter result image:

2. Using Custom Types

When built‑in Grok patterns are insufficient, you can define custom types.

First, you can write an oniguruma regular expression directly:

For example, to capture a 10‑ or 11‑character hexadecimal string and assign it to queue_id:

Second, you can create a custom pattern file. Create a patterns directory and add a file (e.g., postfix) with your definitions:

Reference the custom pattern file in your Logstash configuration:

After adding the custom pattern to the filter configuration, the parsing result looks like:

3. Other Common Built‑in Methods

add_field : After a pattern matches, you can dynamically modify or add new fields using %{fieldName}. Example:

If somefield=dad, Logstash adds a new field foo_dad with the value Hello world, dad.

add_tag : Adds a tag to events that match a filter. Example: