Top Logstash Interview Questions 11‑20: Answers and Practical Configurations

11. HTTP/HTTP_Poller plugins and reading API data

Logstash can retrieve data via HTTP in two ways: the http input opens a listening HTTP port for external services to POST data, while the http_poller input periodically sends requests to a specified API. Example configuration:

input {
  http_poller {
    urls => {
      test_api => {
        method => get
        url => "http://example.com"
      }
    }
    schedule => { cron => "*/5 * * * * UTC" }
    codec => "json"
  }
}

12. Use cases for the Split filter

The Split filter breaks a field containing multiple values or an array into separate events, which is useful for handling batch records. Example:

filter {
  split {
    field => "items"
  }
}

If you need to keep data in a single event, avoid split and use a Ruby filter or custom logic to extract required information.

13. Debugging a Logstash pipeline

Start Logstash with --debug or --log.level debug to get detailed logs.

Inspect Logstash log files (e.g., logstash‑plain.log).

Add an output stdout { codec => rubydebug } to view the processed event structure.

14. Monitoring Logstash performance (Metricbeat)

Use Metricbeat’s logstash module to collect system metrics, event throughput, JVM memory, etc., and visualize them in Kibana.

Alternatively, enable Kibana Monitoring (X‑Pack) and view real‑time processing status in “Stack Monitoring”.

15. Handling Grok parsing failures

Logstash tags failed events with _grokparsefailure.

In the Grok filter, set on_failure => [...] and use conditional logic to route failed events to a separate file or index.

Optionally enable a Dead Letter Queue (DLQ) to store failed events for later analysis.

16. Securing communication between Logstash and Elasticsearch

Enable HTTPS in Elasticsearch (configure certificates and keys).

In Logstash’s Elasticsearch output, set ssl => true and configure cacert for certificate verification.

Enable username/password authentication and role‑based access control (RBAC) to restrict write access.

17. Collecting logs from multiple data sources

Logstash can ingest from several sources by defining multiple input plugins in a single configuration file, e.g.:

input {
  jdbc { ... }   # from a database
  beats { ... }   # receives data from Filebeat
}

Subsequent filters can process events based on type or tags.

18. Sending processed results to multiple destinations

Declare multiple output plugins to write to both Elasticsearch and a local file:

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "my_index"
  }
  file {
    path => "/path/to/output.log"
  }
}

Logstash will send events to all configured outputs in parallel.

19. Differences between Logstash and Elasticsearch Ingest Pipelines

Logstash is an independent service with rich capabilities, supporting many data sources and complex filtering logic.

Ingest Pipeline is a lightweight preprocessing feature built into Elasticsearch, limited to data already sent to ES and suited for simpler use cases.

20. Managing Logstash pipelines in Kibana

If Pipeline Management (X‑Pack) is enabled and properly configured, pipelines can be created and managed via Kibana’s “Management → Logstash Pipelines”. Otherwise, pipelines are typically edited directly on the Logstash host (e.g., pipeline.yml).