Enterprise Log Monitoring System Architecture for Microservice Environments

In large‑scale microservice deployments, logs are often stored locally on each service instance, making it difficult to locate relevant logs for troubleshooting, performance tuning, or business analysis.

The proposed solution is a unified log monitoring system that centralizes log collection, applies filtering and cleaning, and provides a visual interface for monitoring, alerting, and searching.

Key functions include unified log ingestion, filtering/cleaning pipelines, and generation of dashboards and alerts.

The architecture uses Filebeat on every node for log collection, Elastic APM for tracing without code changes, Prometheus for metrics, and Kafka Streams as an ETL layer to filter and enrich logs; Grafana visualizes Prometheus data while Kibana handles APM visual analysis.

Filebeat instances are configured via a backend UI, supporting one‑to‑one or many‑to‑one topic mappings, and also collect MySQL slow‑query logs, error logs, and third‑party service logs such as Nginx.

Elastic APM alone cannot cover all languages (e.g., C) or capture non‑error and business‑specific logs, so Filebeat remains necessary for comprehensive coverage.

Additional capabilities include dynamic filtering rules, windowed collection around error timestamps, per‑service key‑log configuration, slow‑SQL categorization, real‑time business SQL statistics, and adaptive thresholds based on traffic peaks.

Visualization is built primarily with Grafana (integrated with Prometheus and Elasticsearch) and Kibana for APM analysis.