Design and Implementation of a Centralized Database Log Collection and Analysis Platform

In recent years, the company’s application logs were already collected centrally, but database logs remained scattered across host machines and containerized RDS instances, making fault diagnosis difficult.

In 2021 the DBA team developed a database log collection and analysis platform that gathers system logs, error logs, and audit logs from database hosts, normalizes them, and stores them for further processing.

The article outlines the evolution of log analysis through four eras: the Stone Age (manual tools like Excel and shell commands), the Iron Age (custom scripts and simple utilities), the Industrial Age (open‑source and commercial solutions such as the Elastic stack and Splunk), and the Future Age (where machine learning and AI are expected to play a key role).

The platform’s workflow consists of log standardization → log collection → log storage → log analysis → log visualization → alert generation.

Architecture diagrams (Figure 1) show the platform’s layers: application, collection, cache, parsing, storage, and presentation.

Implementation details include the custom log‑pilot collector that captures host, error, and audit logs, pushes them to an isolated Kafka cluster, processes the streams with Flink (including JSON handling for audit logs), and finally indexes the results into Elasticsearch for display in Kibana.

Additional features provide a visual configuration UI for log paths and collection control, as well as alerting integration with the company’s monitoring system (AutoCMP) for error and sensitive audit events.

Figures 2‑4 illustrate the physical instance log collection architecture, dashboard visualizations, and alerting screenshots.

Future work plans to add SQL statement logging for high‑risk operations (e.g., DROP, DELETE) and to further analyze warnings and errors to improve database service quality.