Building a Complete Log Collection and Visualization Pipeline with SpringBoot, Log4j2, Kafka, Filebeat, Logstash, Elasticsearch, and Kibana

In this tutorial, a top‑level architect demonstrates how to build an end‑to‑end log collection and visualization solution for a SpringBoot application.

First, the required servers are prepared, then a SpringBoot demo project is created and configured to use Log4j2 instead of the default logging framework. The pom.xml dependencies and the log4j2.xml configuration are shown.

The demo includes an IndexController that writes info, warn and error logs, and a utility class InputMDC together with NetUtil to inject host, IP and application name into the MDC.

After the application is started, the generated log files app-collector.log and error-collector.log are verified.

Next, Kafka is installed, a broker is configured, and two topics ( app-log-collector and error-log-collector) are created.

Filebeat is then installed and configured to tail the two log files, enrich the events with custom fields, and forward them to Kafka.

A Logstash pipeline is defined to consume the Kafka topics, parse the log lines with Grok, add a time‑based index name, and output the events to Elasticsearch.

Finally, Elasticsearch and Kibana are set up, an index pattern is created, and the collected logs can be visualized in Kibana.