Design and Implementation of an Elastic Stack Based Centralized Log System for 58 Group

Background: With rapid growth of 58 Group's business, log volume increased exponentially, requiring a powerful and reliable log processing system.

Since early 2018, the DB department selected Elastic Stack to handle massive log data, offering collection, transmission, storage, analysis, and alerting capabilities.

Elastic Stack components include Beats, Elasticsearch, Logstash, Kibana, and APM, with ELK as the core suite.

Key components:

Elasticsearch – real‑time full‑text search and analytics engine.

Logstash – collects, parses, and transforms logs from various sources.

Kibana – web UI for visualizing and querying log data.

Beats – lightweight data shippers (Packetbeat, Topbeat, Filebeat, Winlogbeat, Metricbeat, Auditbeat).

Four architectural patterns are described:

1. Simple ELK: Logstash on each node forwards logs directly to Elasticsearch; easy to set up but resource‑heavy and lacks buffering.

2. ELK with message queue (Kafka/Redis): Logstash agents send logs to a queue, providing fault tolerance and better load distribution for larger clusters.

3. ELK with Logstash‑forwarder: Uses Logstash‑forwarder to reduce resource usage on client nodes and secures transport via SSL.

4. ELK with Beats: Replaces Logstash‑forwarder with Beats, offering similar resource usage but greater extensibility and flexibility.

An example MySQL audit log system demonstrates the pipeline: Percona audit plugin → Rsyslog → Filebeat → Kafka → Logstash → Elasticsearch → Kibana for querying.

Conclusion: All logs sent to the company’s Kafka can be ingested into the DB department’s Elasticsearch cluster, enabling unified search, analysis, error tracing, and performance monitoring, with future plans to integrate more Beats components.