How to Optimize Large-Scale Log Systems for Real-Time Monitoring and Scalability

Log System Architecture Benchmark

Developers know that a reliable log platform is essential from initial platform setup to core business operations.

A simple log scenario uses a master/slave application and a shell script to check for errors. As business complexity grows, monitoring a single machine or application is insufficient, especially when isolation prevents access to logs of failing components.

Some applications delete original log files after collection, further complicating maintenance.

Log Processing Flows

Simple flow: Application generates logs → Log rotation based on size/time → Periodic review → Periodic deletion.

Complex flow: Application generates logs → Collection → Transmission → Filtering & transformation → Storage → Analysis & viewing.

Real‑Time vs. Near‑Real‑Time vs. Traceability

Real‑time: Critical user‑facing applications where errors trigger immediate alerts.

Near‑real‑time: Scenarios like project‑management platforms where brief downtime does not affect core outcomes.

Traceability: Historical data retrieval and cross‑time‑dimensional analysis.

Log System Architecture (ELK Mode)

Elasticsearch (ES): Central storage and query engine.

Beats (Filebeat, Topbeat): Lightweight collectors; Filebeat improves Logstash resource usage, Topbeat gathers system metrics.

The platform uses a plugin model (input, output, filter plugins) to keep resource consumption low.

Log System Optimization Ideas

Basic optimization:

Memory: Allocation, garbage collection, caching, locking.

Network: Serialization, compression, protocols.

CPU: Multithreading to increase utilization.

Disk: File merging, defragmentation, disabling unnecessary services.

Platform expansion:

Horizontal scaling via distributed clusters.

Vertical scaling by adding disk and memory.

Data partitioning: Classify logs (error, info, debug) and filter low‑importance levels.

Hotspot handling: Detect time‑based spikes and process them separately.

System degradation: Disable non‑essential features during overload.

Optimization Practice and Results

Key problems addressed:

Low CPU utilization on transmission servers.

Frequent Full GC due to Ruby memory settings.

Disk performance spikes on storage servers.

High‑water‑mark triggers causing service pauses.

Cluster hangs when an ES node fails during heavy load.

Improvements achieved:

Server resource usage reduced; storage saved ~15% per node.

Single‑core throughput increased from ~3k to 15‑18k logs/sec; idle core reaches ~30k logs/sec.

ES protection mechanisms rarely triggered after data sharding.

Extended log retention beyond the original 7‑day limit.

Log Formats

Common fields include UUID, timestamp, host, etc., enabling precise source identification and historical tracing.

Log Solution Overview

Rsyslog can write collected data directly to files or databases. Fluentd offers flexible plugins to forward logs to MongoDB, MySQL, or Elasticsearch.

Overall, the log pipeline consists of three baselines: collection → storage → visualization, with optional transmission and transformation layers based on project needs.