This article describes the shortcomings of a legacy log‑collection architecture, proposes a streamlined real‑time design that centralises configuration in Etcd, uses a single Go‑based logagent to tail files, applies per‑service rate limiting, and forwards logs to Kafka for downstream processing.
This article details the log‑collection difficulties inherent to containerized workloads on the 360 HULK cloud platform and explains how a custom log‑controller, together with Logstash, Qconf, and Qbus2.0, provides dynamic, loss‑free, and source‑aware logging for Kubernetes clusters.
The article details Snowball's challenges with a saturated CDH Hadoop cluster, outlines the limitations of the original Kafka‑based log pipeline, and explains how a comprehensive redesign using FlumeNG, Spillable Memory Channels, and custom HDFS sinks resolves latency, data loss, and high‑load issues while supporting future growth.
This article explores the everyday challenges of operations teams handling system, access, runtime, error, and business logs, outlines the pain points of log collection and standardization, and provides a comprehensive guide to implementing the ELK (Elastic) stack—including Elasticsearch, Logstash, and Kibana—for effective monitoring and analysis.
This article introduces Meitu’s self‑developed log collection system Arachnia, explaining why a custom solution was needed for massive server‑side user‑behavior logs, the key requirements such as reliability and real‑time throughput, and the core architectural mechanisms that address those challenges.
This article explains why service monitoring is essential for large‑scale microservice environments, outlines design principles, core monitoring components, dependency mapping, call‑chain analysis, capacity planning, root‑cause analysis, and presents a practical technical architecture for implementing robust monitoring solutions.
This article explains how LogHub enables real-time log collection and unified management for an e‑commerce takeout platform, covering operational challenges, logstore configuration, user promotion tracking, server and client logging methods, and network access options.
The article presents the architecture, component design, optimizations, and reliability guarantees of Bilibili's Lancer log collection system, a Flume‑based distributed pipeline that handles both real‑time and offline data streams for billions of events daily.
This guide walks you through installing and configuring the ELK stack (Elasticsearch, Logstash, Kibana), setting up the required environment, creating Logback and Logstash configurations, and integrating them with a Spring MVC application to collect and visualize Java logs in Kibana.
Beats are lightweight data shippers in the ELK Stack, and this guide details the purpose, key features, and integration of Filebeat for log collection, Metricbeat for system and service metrics, Packetbeat for network traffic analysis, Winlogbeat for Windows event logs, and Heartbeat for service uptime monitoring.
This article reviews major open‑source log collection tools—Chukwa, Scribe, Flume, Logstash, Kafka, and TT—examining their architectures, strengths, and limitations to help engineers select the most suitable solution for high‑volume, low‑latency data pipelines.
This article presents a practical architecture for centrally collecting dispersed logs from Docker‑based services in a distributed environment by leveraging Flume NG as a non‑intrusive log agent, Kafka as a high‑throughput message bus, and custom sinks to partition logs by service, module, and day.
This article explains how to capture Docker container logs, discusses the challenges of multi‑line log correlation, and presents two approaches—client‑side parsing and server‑side parsing—along with a concrete Flume customization using a DockerLog Java bean.
This guide explains how Oracle's Trace File Analyzer (TFA) simplifies and accelerates log collection across single‑node and RAC databases, covering supported platforms, installation details, workflow, command usage, analysis features, and practical tips for permissions and configuration.
This article discusses various log collection and analysis architectures, comparing solutions such as Flume‑Kafka‑Storm pipelines, Sentry, MongoDB, ELK stack, and Hadoop, and shares practical experiences, advantages, drawbacks, and deployment tips from multiple engineers.
This article provides a comprehensive overview of App Engine as a PaaS solution, analyzes leading platforms such as CloudFoundry and SAE, outlines architectural requirements, and presents practical implementation guidance using open‑source tools like Nginx, Scribe, and Storm for elastic, scalable cloud services.
