How to Set Up ELK Logging for Spring Boot with Filebeat, Elasticsearch, and Kibana

Environment: Spring Boot 2.3.10, Elasticsearch 7.8.0, Kibana 7.8.0, Filebeat 7.8.0.

Install Elasticsearch

Refer to the detailed Spring Boot ELK integration steps.

Install Kibana

Follow the same integration guide.

Install Filebeat

Steps:

Download the required version (7.8.0) from the official site.

Extract the Filebeat archive.

Modify filebeat.yml with the following configuration:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - E:\sts6_projects\logs\es-elk\*.log  # project log directory
  tags: ["file"]
  multiline:
    pattern: ^[0-9]{4}
    negate: true
    match: after
    timeout: 3s
# Index prefix settings
setup.ilm.enabled: false
setup.template.name: "sb"
setup.template.pattern: "sb-log-*"
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
# Output to Elasticsearch
output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "efb-%{+yyyy.MM.dd}"  # daily index
processors:
- script:
    lang: javascript
    id: my_filter
    tag: enable
    source: >
      function process(event) {
        var str = event.Get("message");
        var time = str.split(" ").slice(0,2).join(" ");
        event.Put("start_time", time);
      }
- timestamp:
    field: start_time
    timezone: Asia/Shanghai
    layouts: ['2021-01-01 12:12:05', '2021-01-02 12:12:05.999']
    test: ['2021-01-02 23:59:59']
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~

For additional configuration options, consult the official documentation.

Start Filebeat with the command: filebeat.exe -e -c filebeat.yml Filebeat starts successfully; view the created efb-* index in Kibana.

Spring Boot Project API

@RestController
@RequestMapping("/elk")
public class ElkController {
    private static Logger logger = LoggerFactory.getLogger(ElkController.class);

    @GetMapping("/index")
    public Object index(String info, Integer a) {
        logger.info("You entered: {}", info);
        if (a == 0) {
            logger.error("An error occurred: {}", new RuntimeException("Invalid number"));
        }
        return "success";
    }
}

After launching the service, Kibana displays the startup logs, normal request logs, and error logs generated by the controller.

The logging pipeline is now fully operational.