Enterprise Log Monitoring System Architecture and Implementation

Background : In large‑scale microservice environments, hundreds of services generate logs that are stored locally, making it difficult to locate logs for troubleshooting, performance tuning, and business analysis. Centralizing log collection and processing is essential for effective operations.

Solution Overview : The proposed log monitoring system aggregates logs from all services, filters and cleans them, and provides visual dashboards, alerts, and searchable interfaces for both operations and development teams.

Key Functional Flow :

Log agents are deployed on each service node to collect logs in real time.

Collected logs are sent to a unified log collection service where they are filtered, cleaned, and made available through visual dashboards and alerting mechanisms.

Architecture Details :

Log collection uses filebeat agents, each configured via a backend UI; topics can be one‑to‑one or many‑to‑one based on volume.

In addition to business logs, MySQL slow‑query/error logs and third‑party logs (e.g., Nginx) are also collected.

Elastic APM agents capture call stacks, HTTP traces, SQL statements, and process metrics without requiring code changes.

Prometheus gathers server‑level metrics.

Kafka Streams (Log Streams) performs ETL filtering, dynamic rule configuration, and log enrichment.

Visualization is handled by Grafana (for Prometheus and Elasticsearch data) and Kibana (for APM analysis).

Filtering and Resource Management :

All logs are initially ingested into a Kafka cluster with a short retention window (e.g., one hour) to limit storage costs.

Dynamic filtering rules allow collection of error‑level logs by default, with configurable windows to capture surrounding info‑level logs.

Each service can define up to 100 key logs for full collection.

Slow‑SQL logs can be further filtered by business category.

During peak periods, logs are filtered based on service weight, log level, and per‑service limits.

Log indices are generated per service and per log level (debug, info, error, custom keywords) with date suffixes to match developers' existing habits.

Visualization : Grafana dashboards display metrics from Prometheus and Elasticsearch, while Kibana provides APM visual analysis. Screenshots in the original article illustrate the UI components and data flow diagrams.

Conclusion : By centralizing log collection, applying intelligent filtering, and leveraging open‑source observability tools, the system reduces resource consumption, improves troubleshooting efficiency, and provides actionable insights for both operations and development teams.