BestHub
Sign in
Tagged articles

log collection

113 articles · Page 1 of 2
Mingyi World Elasticsearch
Mingyi World Elasticsearch
May 31, 2026 · Operations

How to Collect Easysearch Logs with Filebeat OSS: A Step‑by‑Step Guide

This guide walks through selecting Filebeat OSS 7.10.2, preparing Ubuntu 20.04, uploading and extracting the package, configuring filebeat.yml for Easysearch log paths, creating an index template, starting Filebeat, verifying data ingestion, and applying production‑grade recommendations such as systemd service setup and ILM policies.

EasysearchElasticsearchUbuntu
0 likes · 11 min read
How to Collect Easysearch Logs with Filebeat OSS: A Step‑by‑Step Guide
Alibaba Cloud Observability
Alibaba Cloud Observability
Mar 9, 2026 · Cloud Native

How LoongCollector’s One‑Time File Collection Simplifies Bulk Log Migration

LoongCollector introduces a One‑Time file collection mode that scans matching files once, records a snapshot, and exits, enabling efficient historic log migration, data back‑fill, and temporary batch processing while providing fine‑grained checkpoints, execution windows, and throttling controls to avoid quota issues and ensure reliable completion.

CheckpointData Migrationlog collection
0 likes · 12 min read
How LoongCollector’s One‑Time File Collection Simplifies Bulk Log Migration
Code Ape Tech Column
Code Ape Tech Column
Jan 7, 2026 · Operations

Is Syslog Still the Best Choice for Simple, Low‑Cost Log Collection?

This article explains why the decades‑old Syslog protocol remains relevant in modern microservice and cloud‑native environments, detailing its structure, advantages, typical use cases, step‑by‑step server setup with rsyslog, command‑line analysis techniques, integration with Java applications, and a comparison with ELK/Loki solutions.

log analysislog collectionrsyslog
0 likes · 20 min read
Is Syslog Still the Best Choice for Simple, Low‑Cost Log Collection?
Alibaba Cloud Observability
Alibaba Cloud Observability
Jul 28, 2025 · Information Security

How to Standardize and Collect Enterprise Firewall Logs with LoongCollector

This guide explains why modern enterprises need unified log collection for diverse security devices, introduces LoongCollector's flexible configuration for parsing and standardizing firewall logs from WAF, FortiGate, and Palo Alto, and provides step‑by‑step examples, code snippets, and visual results to enable efficient threat analysis and compliance.

Log StandardizationLoongCollectorfirewall
0 likes · 17 min read
How to Standardize and Collect Enterprise Firewall Logs with LoongCollector
FunTester
FunTester
May 28, 2025 · Cloud Native

Extending Automated Thread Dumps: Log Collection, Resource Monitoring, Chaos Engineering, Performance Analysis, and Environment Cleanup

The article explores how automated thread dumps can be expanded into multiple testing scenarios—including log collection, resource monitoring, fault injection, performance result analysis, and environment cleanup—by leveraging Kubernetes APIs, Prometheus, Chaos Mesh, and scripting tools to improve efficiency, observability, and system resilience.

AutomationKubernetesResource Monitoring
0 likes · 9 min read
Extending Automated Thread Dumps: Log Collection, Resource Monitoring, Chaos Engineering, Performance Analysis, and Environment Cleanup
Alibaba Cloud Observability
Alibaba Cloud Observability
May 19, 2025 · Cloud Native

How LoongCollector Transforms Log Collection with High‑Performance Pipelines

LoongCollector, the 2025 evolution of iLogtail, introduces a fully redesigned pipeline architecture, hot‑reload isolation, significant CPU and memory reductions, and advanced monitoring, delivering up to 80% higher log‑collection throughput for cloud‑native environments while ensuring seamless upgrades and multi‑region fault tolerance.

Observabilitylog collectionpipeline
0 likes · 14 min read
How LoongCollector Transforms Log Collection with High‑Performance Pipelines
Alibaba Cloud Native
Alibaba Cloud Native
May 15, 2025 · Cloud Native

Why LoongCollector Beats iLogtail and Open‑Source Log Agents by Up to 10×

In a series of controlled benchmarks on an Alibaba Cloud ECS instance, the next‑generation LoongCollector log‑collection agent demonstrated dramatically higher throughput, lower CPU usage, and more efficient memory consumption than its predecessor iLogtail and popular open‑source alternatives such as FluentBit, Vector, and Filebeat across multiple log formats and traffic levels.

Cloud NativeLoongCollectorPerformance Benchmark
0 likes · 13 min read
Why LoongCollector Beats iLogtail and Open‑Source Log Agents by Up to 10×
Alibaba Cloud Developer
Alibaba Cloud Developer
May 14, 2025 · Cloud Native

How LoongCollector Redefines Log Collection: Performance, Flexibility, and Zero‑Downtime Upgrades

LoongCollector, the 2025 evolution of iLogtail, delivers a comprehensive overhaul of log collection with high‑performance C++/Go pipelines, dynamic plugin combos, hot‑reload isolation, adaptive network throttling, enhanced tag handling, seamless migration, and robust monitoring for cloud‑native environments, achieving up to 80% higher throughput while cutting CPU usage by 35% and memory by 10%.

KubernetesPerformance Optimizationlog collection
0 likes · 15 min read
How LoongCollector Redefines Log Collection: Performance, Flexibility, and Zero‑Downtime Upgrades
Alibaba Cloud Observability
Alibaba Cloud Observability
May 12, 2025 · Cloud Native

Building a Cost‑Effective, High‑Availability Global Log Collection with Alibaba Cloud SLS

This article explains how enterprises can overcome the challenges of globally collecting logs by using Alibaba Cloud's high‑performance iLogtail/LoongCollector agents, selecting optimal network paths, applying cost‑saving strategies such as CloudLens diagnostics and LZ4 compression, and configuring multi‑region dual‑write for reliable, low‑cost log delivery.

Alibaba CloudLoongCollectorSLS
0 likes · 24 min read
Building a Cost‑Effective, High‑Availability Global Log Collection with Alibaba Cloud SLS
Alibaba Cloud Native
Alibaba Cloud Native
May 8, 2025 · Cloud Native

Optimizing Global Log Collection with Alibaba Cloud iLogtail and LoongCollector

This guide explains how enterprises can reliably and cost‑effectively gather logs from distributed overseas environments into Alibaba Cloud Log Service (SLS) by selecting the right network path, leveraging iLogtail/LoongCollector agents, applying compression, filtering, multi‑region routing, and using CloudLens for monitoring and cost control.

Alibaba CloudCloud NativeLoongCollector
0 likes · 21 min read
Optimizing Global Log Collection with Alibaba Cloud iLogtail and LoongCollector
Alibaba Cloud Observability
Alibaba Cloud Observability
Apr 1, 2025 · Cloud Native

How We Boosted Multi-line Log Collection Speed from 90 MB/s to 350 MB/s

This article details a real‑world case where massive multi‑line log volumes overwhelmed iLogtail, explains the performance bottlenecks caused by full‑line regex matching, describes the switch to prefix‑only matching and IngestProcessor, and shows how these changes lifted throughput from under 100 MB/s to over 300 MB/s while halving CPU usage.

Performance OptimizationiLogtaillog collection
0 likes · 15 min read
How We Boosted Multi-line Log Collection Speed from 90 MB/s to 350 MB/s
Didi Tech
Didi Tech
Mar 27, 2025 · Operations

Performance Optimization and Architecture of iLogTail for High‑Scale Log Collection

Didi replaced its legacy agent with Alibaba’s open‑source iLogTail, re‑architected it to use a shared thread‑pool and SIMD‑accelerated parsing, rewrote critical plugins in C++ and added robust Kafka retry logic, achieving over twice the throughput while cutting CPU usage by more than half and maintaining near‑zero latency at massive scale.

C++KafkaPerformance Optimization
0 likes · 10 min read
Performance Optimization and Architecture of iLogTail for High‑Scale Log Collection
Sanyou's Java Diary
Sanyou's Java Diary
Mar 6, 2025 · Backend Development

8 Real-World MQ Use Cases Every Backend Engineer Should Know

This article explores eight practical scenarios for using message queues—such as asynchronous processing, service decoupling, traffic shaping, delayed tasks, log collection, distributed transactions, remote calls, and broadcast notifications—providing code examples with RabbitMQ, RocketMQ, and Kafka to illustrate each pattern.

Broadcast NotificationsDelayed TasksMessage Queue
0 likes · 15 min read
8 Real-World MQ Use Cases Every Backend Engineer Should Know
Alibaba Cloud Developer
Alibaba Cloud Developer
Dec 3, 2024 · Operations

How to Boost Logtail Multiline Log Collection Speed by Up to 7×

This article investigates why enabling line‑prefix regex for multiline logs slows Logtail down, explains the underlying regex matching mechanism, and demonstrates how switching from boost::regex_match to boost::regex_search with proper flags can dramatically improve collection throughput, achieving a seven‑fold speed increase.

boost::regexlog collectionlogtail
0 likes · 10 min read
How to Boost Logtail Multiline Log Collection Speed by Up to 7×
Top Architect
Top Architect
Nov 4, 2024 · Operations

Centralized Log Collection and Monitoring with Filebeat and Graylog

This article explains how to use Filebeat as a lightweight log shipper together with Graylog for centralized log aggregation, storage, and analysis, covering installation, configuration files, Docker deployment, Spring Boot integration, and query techniques for efficient operations monitoring.

DockerElasticsearchGraylog
0 likes · 18 min read
Centralized Log Collection and Monitoring with Filebeat and Graylog
Alibaba Cloud Native
Alibaba Cloud Native
Oct 11, 2024 · Cloud Native

Can iLogtail Replace Logstash? A Deep Dive into Performance and Architecture

This article examines the traditional ELK stack, compares iLogtail with Filebeat and Logstash in real‑world performance tests, analyzes why iLogtail could not previously replace Logstash, and presents five concrete engineering solutions that enable iLogtail to become a viable, high‑performance alternative for log collection and processing.

Cloud NativeELKObservability
0 likes · 12 min read
Can iLogtail Replace Logstash? A Deep Dive into Performance and Architecture
Alibaba Cloud Observability
Alibaba Cloud Observability
Oct 9, 2024 · Cloud Native

How iLogtail Evolved Over 13 Years to Lead Cloud‑Native Observability

iLogtail, a lightweight log collector, has transformed over 13 years from a simple log‑gathering tool into a full‑stack, cloud‑native observability platform, introducing Go plugins, high‑performance C++ pipelines, SPL processing, modular architecture, and advanced self‑monitoring, reflecting broader trends in data collection technology.

ObservabilityPerformance Optimizationlog collection
0 likes · 22 min read
How iLogtail Evolved Over 13 Years to Lead Cloud‑Native Observability
Alibaba Cloud Native
Alibaba Cloud Native
Sep 26, 2024 · Cloud Native

How iLogtail Evolved: From Simple Log Collector to Cloud‑Native Observability Platform

This article chronicles iLogtail's 13‑year journey—from its 2013 inception as a basic log collector to a fully open‑source, cloud‑native observability platform—highlighting technical milestones, emerging trends in log agents, architectural innovations, performance breakthroughs, and future directions.

Cloud NativeObservabilityiLogtail
0 likes · 21 min read
How iLogtail Evolved: From Simple Log Collector to Cloud‑Native Observability Platform
Alibaba Cloud Developer
Alibaba Cloud Developer
Sep 26, 2024 · Cloud Native

How iLogtail Simplifies Container Log Collection Across Kubernetes Runtimes

This article explains Kubernetes container runtimes, the CRI and OCI standards, compares Docker, containerd, CRI‑O and runc, outlines the challenges of log collection in K8s, and shows how open‑source collectors like Filebeat and Fluent Bit as well as iLogtail's daemonset and sidecar modes provide flexible, enriched logging solutions.

Fluent Bitcontainer runtimefilebeat
0 likes · 32 min read
How iLogtail Simplifies Container Log Collection Across Kubernetes Runtimes
Alibaba Cloud Native
Alibaba Cloud Native
Aug 21, 2024 · Cloud Native

What Drives iLogtail Adoption? Insights from a Two‑Year Community Survey

A two‑year community survey of the open‑source iLogtail collector reveals that high performance, container‑friendly design, extensive plugin ecosystem, and strong Kubernetes integration drive widespread production use, while users request better documentation, a more polished ConfigServer tool, and clearer contribution pathways.

Cloud NativeObservabilitylog collection
0 likes · 10 min read
What Drives iLogtail Adoption? Insights from a Two‑Year Community Survey
Architect
Architect
Feb 28, 2024 · Cloud Native

Lightweight Kubernetes Log Collection with Loki: Deployment and Configuration Guide

This article provides a comprehensive, lightweight solution for collecting Kubernetes logs using Grafana Loki, covering its advantages, component comparison, deployment modes (All‑In‑One, microservices, bare‑metal), required configuration files, ConfigMap and PersistentVolume setup, Promtail installation, Helm deployment, and common troubleshooting steps.

Kuberneteshelmlog collection
0 likes · 20 min read
Lightweight Kubernetes Log Collection with Loki: Deployment and Configuration Guide
Alibaba Cloud Native
Alibaba Cloud Native
Feb 20, 2024 · Cloud Native

What’s New in iLogtail 2.0? A Deep Dive into the Updated Pipeline Architecture

iLogtail 2.0 replaces the monolithic, file‑oriented design of its predecessor with a modular pipeline configuration, new input/processor/output plugins, a refreshed API, SPL processing, finer‑grained parsing controls, nanosecond‑level timestamps, enhanced observability, and performance improvements, while providing compatibility guidance for both commercial and open‑source editions.

APICloud NativeObservability
0 likes · 17 min read
What’s New in iLogtail 2.0? A Deep Dive into the Updated Pipeline Architecture
MaGe Linux Operations
MaGe Linux Operations
Feb 10, 2024 · Backend Development

Mastering the Sidecar Pattern: Log Collection, Request Forwarding, and Interception in Kubernetes

This article explains the sidecar concept, compares it with SDK approaches, and provides detailed Kubernetes examples—including a log‑collection sidecar, a request‑forwarding sidecar, and an HTTP‑intercepting sidecar—complete with YAML manifests and Rust and Scala code to demonstrate implementation and deployment.

KubernetesRustScala
0 likes · 9 min read
Mastering the Sidecar Pattern: Log Collection, Request Forwarding, and Interception in Kubernetes
Architect
Architect
Dec 30, 2023 · Big Data

Designing a Scalable Log Collection Agent: Lessons from Vivo’s Bees‑Agent

This article details the end‑to‑end design of Vivo’s custom log‑collection agent, covering file discovery with inotify, unique file identification using inode and content hash, real‑time reading via RandomAccessFile, checkpointing, Kafka integration, offline HDFS ingestion, resource throttling, and platform‑wide management, while comparing it with open‑source alternatives.

Agent DesignBig DataKafka
0 likes · 26 min read
Designing a Scalable Log Collection Agent: Lessons from Vivo’s Bees‑Agent
Code Ape Tech Column
Code Ape Tech Column
Dec 12, 2023 · Operations

Centralized Log Collection with Filebeat and Graylog

This article explains how to use Filebeat together with Graylog to collect, ship, store, and analyze logs from multiple environments, covering tool introductions, configuration files, Docker deployment, Spring Boot integration, and practical search syntax for effective log monitoring.

ElasticsearchGraylogLogging
0 likes · 20 min read
Centralized Log Collection with Filebeat and Graylog
DataFunTalk
DataFunTalk
Jul 9, 2023 · Operations

Building High‑Performance Observability Data Pipelines with Vector and Honghu

This article explains the concepts and importance of observability, introduces the Vector data‑pipeline tool and its architecture, demonstrates how to configure sources, transforms and sinks, and shows how to integrate Vector with the Honghu platform to build a complete, real‑time monitoring solution for modern distributed systems.

Big DataHonghuMonitoring
0 likes · 33 min read
Building High‑Performance Observability Data Pipelines with Vector and Honghu
JD Cloud Developers
JD Cloud Developers
Apr 20, 2023 · Operations

How to Build a Cost‑Effective, High‑Throughput Log Collection System with ClickHouse

This article examines the challenges of scaling log storage and retrieval for high‑traffic services, analyzes the cost and performance limits of traditional ELK‑based pipelines, and presents a streamlined, UDP‑driven architecture using ClickHouse that dramatically reduces hardware expenses while handling hundreds of gigabytes per second.

ClickHouseUDPbackend infrastructure
0 likes · 16 min read
How to Build a Cost‑Effective, High‑Throughput Log Collection System with ClickHouse
Efficient Ops
Efficient Ops
Apr 17, 2023 · Operations

Mastering Container Log Collection in Kubernetes: Strategies and Best Practices

This article explains how container log collection in Kubernetes differs from traditional host logging, outlines common deployment methods such as DaemonSet and Sidecar, compares log storage options, and offers practical guidance on handling stdout and file‑based logs for reliable operations.

DaemonSetKubernetesSidecar
0 likes · 12 min read
Mastering Container Log Collection in Kubernetes: Strategies and Best Practices
dbaplus Community
dbaplus Community
Mar 13, 2023 · Cloud Native

From Bare Metal to Cloud‑Native: How Zhuanzhuan Reinvented Log Collection

This article traces Zhuanzhuan's evolution of log collection—from a bare‑metal scribe + flume pipeline, through a container‑aware log‑pilot solution, to a cloud‑native filebeat and fb‑advisor architecture—detailing the motivations, technical designs, performance gains, and trade‑offs of each stage.

Operationscontainerfilebeat
0 likes · 12 min read
From Bare Metal to Cloud‑Native: How Zhuanzhuan Reinvented Log Collection
Zhuanzhuan Tech
Zhuanzhuan Tech
Feb 1, 2023 · Cloud Native

Evolution of Log Collection at ZuanZuan: From Bare Metal to Cloud‑Native Era

This article traces ZuanZuan's log‑collection journey from the bare‑metal era through containerization to a cloud‑native solution, detailing the challenges, custom developments like log‑pilot, ByteCompass, and fb‑advisor, and the performance gains achieved with each transition.

Cloud Nativecontainerizationfilebeat
0 likes · 12 min read
Evolution of Log Collection at ZuanZuan: From Bare Metal to Cloud‑Native Era
Efficient Ops
Efficient Ops
Jan 17, 2023 · Operations

Mastering Observability Data Collection in Kubernetes with iLogtail

This article explains the types and value of observability data, the characteristics and data‑collection requirements of Kubernetes deployments, the challenges of large‑scale log, metric and trace ingestion, and practical solutions such as DaemonSet vs Sidecar, ConfigServer, CRD automation, serverless support, and eBPF integration.

iLogtaillog collection
0 likes · 20 min read
Mastering Observability Data Collection in Kubernetes with iLogtail
NetEase Yanxuan Technology Product Team
NetEase Yanxuan Technology Product Team
Jan 9, 2023 · Operations

Loggie: A High-Performance Log Collection Agent System Design and Implementation

Loggie is a cloud-native, Go-based log-collection agent that replaces Filebeat and Flume by using a micro-kernel producer-consumer architecture with hot-swappable pipelines, achieving 2 GB/s read speeds, 1.6‑2.6× higher throughput while using only a quarter of the CPU, and providing built-in observability, reliability, and latency monitoring for large-scale enterprise deployments.

GoMonitoringOperations
0 likes · 16 min read
Loggie: A High-Performance Log Collection Agent System Design and Implementation
Architecture Digest
Architecture Digest
Dec 2, 2022 · Big Data

Design and Implementation of Vivo's Bees Log Collection Agent

This article presents the design principles, core techniques, and practical solutions of Vivo's self‑developed Bees log collection agent, covering file discovery, unique identification, real‑time and offline ingestion, checkpointing, resource control, platform management, and a comparison with open‑source alternatives.

Agent DesignJavaKafka
0 likes · 25 min read
Design and Implementation of Vivo's Bees Log Collection Agent
High Availability Architecture
High Availability Architecture
Nov 30, 2022 · Big Data

Design and Implementation of Vivo's Bees Log Collection Agent

This article presents the design principles, core features, and implementation details of Vivo's self‑developed Bees log collection agent, covering file discovery, unique identification, real‑time and offline ingestion, resource control, platform management, and comparisons with open‑source solutions.

HDFSJavaKafka
0 likes · 22 min read
Design and Implementation of Vivo's Bees Log Collection Agent
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 28, 2022 · Big Data

Comprehensive Guide to Big Data Interview Topics: Log Collection, Data Synchronization, Offline Development, Real‑time Technology, Data Services, and Data Mining

This article provides an extensive overview of big‑data interview subjects, covering browser and mobile log collection methods, data synchronization techniques (batch, real‑time, sharding), offline data development platforms, streaming architectures, data service evolution, performance optimization, and data‑mining layers and applications.

Big DataData synchronizationStreaming
0 likes · 17 min read
Comprehensive Guide to Big Data Interview Topics: Log Collection, Data Synchronization, Offline Development, Real‑time Technology, Data Services, and Data Mining
vivo Internet Technology
vivo Internet Technology
Nov 23, 2022 · Big Data

Design and Implementation of Vivo's Bees Log Collection Agent

Vivo’s Bees‑agent is a custom, lightweight log‑collection service that discovers rotating files via inotify, uniquely identifies them with inode and hash signatures, supports real‑time and offline ingestion to Kafka and HDFS, offers checkpoint‑resume, resource isolation, rich metrics, and a centralized management platform, outperforming open‑source collectors in latency, memory usage, and scalability.

Agent DesignHDFSJava
0 likes · 24 min read
Design and Implementation of Vivo's Bees Log Collection Agent
NetEase Yanxuan Technology Product Team
NetEase Yanxuan Technology Product Team
Oct 17, 2022 · Big Data

Design and Evolution of a Unified Log Platform

NetEase Yanxuan built a unified log platform that consolidates collection, delivery, processing, storage, analysis, and alerting for near‑real‑time and offline logs, using Flume, a Golang‑based Loggie agent, Kafka, Flink, HBase and Elasticsearch to ensure high performance, data‑quality, container‑native isolation, full‑link traceability, and automated scaling, with Loggie now open‑sourced.

log collectionlog managementlog platform
0 likes · 38 min read
Design and Evolution of a Unified Log Platform
Wukong Talks Architecture
Wukong Talks Architecture
Aug 24, 2022 · Cloud Native

Overview of ZuanZuan Cloud Platform: Architecture, Image Management, Release Upgrade, Container Monitoring, and Log Collection

The article introduces the ZuanZuan cloud platform, detailing its overall architecture, image management workflow that abstracts Dockerfiles, release‑upgrade strategies with custom controllers, container monitoring evolution to Prometheus, and log‑collection mechanisms that handle large Java‑based log volumes.

Image ManagementMonitoringcloud-native
0 likes · 8 min read
Overview of ZuanZuan Cloud Platform: Architecture, Image Management, Release Upgrade, Container Monitoring, and Log Collection
Alibaba Cloud Developer
Alibaba Cloud Developer
Aug 23, 2022 · Cloud Native

Why iLogtail’s Open‑Source Cloud‑Native Agent Is Redefining Observability

This article explores the open‑source release of Alibaba Cloud's iLogtail, detailing its lightweight, high‑performance design, multi‑tenant isolation, plugin architecture, Kubernetes integration, and the differences between its enterprise and community editions, while highlighting its role in modern observability pipelines.

KubernetesMonitoringlog collection
0 likes · 27 min read
Why iLogtail’s Open‑Source Cloud‑Native Agent Is Redefining Observability
DataFunTalk
DataFunTalk
Jul 31, 2022 · Big Data

Design, Evolution, and Optimization of NetEase's Log Collection and Transmission Service (Datastream‑NG)

This article presents a comprehensive overview of NetEase's log collection and transmission platform, detailing its evolution from 2011 to the current Datastream‑NG architecture, the system's design goals, core component optimizations, operational monitoring, and future plans for intelligent scaling and diagnostics.

Big DataCloud NativeData Streaming
0 likes · 23 min read
Design, Evolution, and Optimization of NetEase's Log Collection and Transmission Service (Datastream‑NG)
DevOps Cloud Academy
DevOps Cloud Academy
Jun 17, 2022 · Operations

Fluentd Installation, Configuration, and Usage Guide

This guide explains how to install Fluentd via Docker or manual methods, configure the environment (NTP, file descriptors, kernel parameters), use the td-agent package, manage services with systemd, send test logs, and understand Fluentd's event lifecycle, filters, labels, and configuration directives.

DockerFluentdlog collection
0 likes · 27 min read
Fluentd Installation, Configuration, and Usage Guide
JD Tech
JD Tech
Apr 8, 2022 · Big Data

Designing a High‑Throughput Log Collection System with ClickHouse, UDP and Compression

The article analyses the massive cost and performance challenges of traditional log‑storage pipelines at JD.com, then proposes a streamlined architecture that eliminates disk and MQ stages, uses in‑memory buffering, UDP transport, Snappy/ZSTD compression, and ClickHouse storage to achieve multi‑gigabyte per‑second throughput with over 70% hardware cost reduction.

ClickHousedistributed systemshigh-throughput
0 likes · 15 min read
Designing a High‑Throughput Log Collection System with ClickHouse, UDP and Compression
Code Ape Tech Column
Code Ape Tech Column
Apr 8, 2022 · Operations

Filebeat + Graylog: A Complete Guide to Log Collection, Processing, and Visualization

This article introduces the Filebeat log‑shipping tool and the Graylog log‑management platform, explains their architectures and workflows, provides detailed configuration examples for Filebeat inputs, modules, and Graylog components, and walks through deployment using Docker, Docker‑Compose, and native package installations.

DockerGraylogfilebeat
0 likes · 15 min read
Filebeat + Graylog: A Complete Guide to Log Collection, Processing, and Visualization
Practical DevOps Architecture
Practical DevOps Architecture
Feb 16, 2022 · Operations

Understanding Filebeat Harvester, Prospector, and Configuration for System Log Collection

This article explains how Filebeat’s harvester and prospector components read and forward system logs, maintain file offsets in a registry, and provides a sample YAML configuration for collecting logs from a specified file and sending them to Elasticsearch, illustrating key operational concepts for log management.

ELKlog collectionsystem logs
0 likes · 3 min read
Understanding Filebeat Harvester, Prospector, and Configuration for System Log Collection
Efficient Ops
Efficient Ops
Jan 9, 2022 · Operations

How to Collect Nginx Access and Error Logs with Filebeat, Logstash, and Rsyslog

This guide demonstrates multiple ways to gather Nginx access and error logs—directly with Filebeat to Elasticsearch, via Filebeat to Logstash then Elasticsearch, and using rsyslog to forward logs to Logstash—providing step‑by‑step configurations, code snippets, and visual illustrations for each method.

ElasticsearchLogstashNGINX
0 likes · 9 min read
How to Collect Nginx Access and Error Logs with Filebeat, Logstash, and Rsyslog
HomeTech
HomeTech
Jan 6, 2022 · Operations

Design and Implementation of a Centralized Database Log Collection and Analysis Platform

This article describes the background, architecture, and implementation of a centralized database log collection and analysis platform built in 2021, detailing how logs from hosts, containers, and databases are normalized, streamed through Kafka, processed with Flink, stored in Elasticsearch, visualized with Kibana, and extended with alerting and configuration management to improve fault diagnosis and lay the groundwork for future AI‑driven operations.

Big DataKibanaMonitoring
0 likes · 5 min read
Design and Implementation of a Centralized Database Log Collection and Analysis Platform
Architecture Digest
Architecture Digest
Dec 23, 2021 · Operations

Using Filebeat and Graylog for Centralized Log Collection and Monitoring

This article explains how to deploy and configure Filebeat and Graylog for centralized log collection, covering installation methods, configuration files, Docker deployment, input modules, pipelines, and practical examples for efficiently gathering and analyzing logs across multiple environments.

DockerGraylogMonitoring
0 likes · 15 min read
Using Filebeat and Graylog for Centralized Log Collection and Monitoring
dbaplus Community
dbaplus Community
Dec 14, 2021 · Backend Development

How to Build a Cost‑Effective High‑Throughput Log Collection System with ClickHouse and UDP

This article analyzes the challenges of massive log storage and retrieval, calculates the bandwidth and hardware costs of traditional pipelines, and presents a streamlined architecture that uses in‑memory buffering, UDP transport, compression, and ClickHouse to achieve petabyte‑scale throughput while cutting storage costs by over 75%.

UDPbackendhigh-throughput
0 likes · 16 min read
How to Build a Cost‑Effective High‑Throughput Log Collection System with ClickHouse and UDP
Alibaba Cloud Developer
Alibaba Cloud Developer
Dec 9, 2021 · Operations

How Alibaba’s iLogtail Revolutionizes Observability at Massive Scale

This article details the open‑source release of Alibaba’s iLogtail, explains the observability challenges it solves, traces its evolution through the Feitian 5K, Alibaba Group, and cloud‑native stages, highlights its high‑performance, stability, and multi‑tenant features, and outlines the community‑driven open‑source goals alongside a brief Impala overview.

Performancelog collection
0 likes · 15 min read
How Alibaba’s iLogtail Revolutionizes Observability at Massive Scale
Top Architect
Top Architect
Dec 3, 2021 · Operations

Centralized Log Collection with Filebeat and Graylog

This article explains how to use Filebeat together with Graylog to collect, process, and visualize logs from multiple services and environments, covering tool introductions, configuration files, component details, deployment methods, and practical code examples.

DockerELKGraylog
0 likes · 19 min read
Centralized Log Collection with Filebeat and Graylog
JD Retail Technology
JD Retail Technology
Nov 22, 2021 · Backend Development

Designing a High‑Performance Log Collection System with UDP, Compression, and ClickHouse

The article analyzes the high cost and scalability challenges of traditional log collection pipelines and proposes a streamlined architecture that uses in‑memory buffering, UDP transport, aggressive compression, and ClickHouse storage to achieve massive throughput while drastically reducing hardware and operational expenses.

ClickHouseMonitoringUDP
0 likes · 15 min read
Designing a High‑Performance Log Collection System with UDP, Compression, and ClickHouse
58 Tech
58 Tech
Nov 11, 2021 · Frontend Development

Cross‑Platform Mini Program Log Collection SDK: Design, Implementation, and Practice

This article introduces a lightweight, cross‑platform SDK for mini‑programs that unifies log collection—including exception, normal, and performance data—by intercepting App, Page and platform‑specific APIs, handling multi‑environment differences, and providing flexible usage via npm or single‑file integration.

Mini Programcross‑platformfrontend
0 likes · 15 min read
Cross‑Platform Mini Program Log Collection SDK: Design, Implementation, and Practice
360 Tech Engineering
360 Tech Engineering
Oct 15, 2021 · Operations

Log Collection Architecture Using Filebeat, Logstash, and Kafka

This article describes a lightweight, resource‑efficient log collection solution that combines Filebeat agents, optional Logstash aggregation, and Kafka transport, detailing configuration choices, meta‑persistence, back‑pressure mechanisms, monitoring setup, and deployment architecture for reliable at‑least‑once delivery.

LogstashMonitoringOperations
0 likes · 14 min read
Log Collection Architecture Using Filebeat, Logstash, and Kafka
Efficient Ops
Efficient Ops
Oct 11, 2021 · Operations

Collect Nginx Access & Error Logs with Filebeat, Logstash, and Rsyslog

This guide walks through three practical methods for harvesting Nginx access and error logs—directly with Filebeat to Elasticsearch, via Filebeat‑Logstash‑Elasticsearch pipeline, and using Rsyslog to forward logs to Logstash—complete with configuration snippets and visual illustrations.

ElasticsearchLogstashNGINX
0 likes · 8 min read
Collect Nginx Access & Error Logs with Filebeat, Logstash, and Rsyslog
MaGe Linux Operations
MaGe Linux Operations
Jul 3, 2021 · Backend Development

How to Build a Go Log Collector with etcd, Context, and Kafka Integration

This article walks through redesigning a Go‑based log‑collection framework, introducing etcd for distributed configuration, demonstrating context for timeout and data propagation, and showing how to integrate Kafka consumers while improving concurrency handling and adding rate‑limiting mechanisms.

ContextKafkabackend
0 likes · 16 min read
How to Build a Go Log Collector with etcd, Context, and Kafka Integration
Liangxu Linux
Liangxu Linux
May 13, 2021 · Operations

Build a Real‑Time Log Collector with Java and tail – Why -F Beats -f

This article shows how to use the Unix tail command together with a simple Java program to collect live logs, explains the crucial difference between tail -f and tail -F during file rotation, and provides practical steps and commands for reliable log monitoring.

JavaReal-time loggingfile rotation
0 likes · 8 min read
Build a Real‑Time Log Collector with Java and tail – Why -F Beats -f
ITFLY8 Architecture Home
ITFLY8 Architecture Home
Mar 23, 2021 · Cloud Native

How to Build a Scalable Cloud‑Native Log Collection System with Filebeat and Custom Controllers

This article explains the challenges of container log collection in Kubernetes, evaluates log‑agent options, details the design of a custom Filebeat‑based controller architecture, shares performance tuning with Golang, and outlines monitoring and future extensions for a robust cloud‑native logging solution.

Kubernetescloud-nativefilebeat
0 likes · 14 min read
How to Build a Scalable Cloud‑Native Log Collection System with Filebeat and Custom Controllers
ITFLY8 Architecture Home
ITFLY8 Architecture Home
Jan 31, 2021 · Cloud Native

Solving Cloud‑Native Log Collection: NetEase Lightboat’s Architecture & Insights

This article explains how NetEase’s Lightboat micro‑service platform tackles the challenges of log collection in cloud‑native Kubernetes environments by designing a custom controller, selecting Filebeat as the agent, integrating it via DaemonSet, extending its functionality, and applying Golang performance‑tuning techniques to achieve efficient, observable logging.

KubernetesPerformance Optimizationcloud-native
0 likes · 13 min read
Solving Cloud‑Native Log Collection: NetEase Lightboat’s Architecture & Insights
21CTO
21CTO
Jan 16, 2021 · Backend Development

How to Build a Go‑Based Log Collection System with etcd, Context, and Kafka

This article walks through designing and implementing a Go log‑collection agent that uses etcd for configuration storage, context for timeout and metadata handling, and Kafka for message consumption, complete with code examples, setup instructions, and a rate‑limiting utility.

ContextEtcdGo
0 likes · 16 min read
How to Build a Go‑Based Log Collection System with etcd, Context, and Kafka
Architect
Architect
Dec 23, 2020 · Operations

Design and Evaluation of Log Collection Agents: Flume vs Filebeat

This article analyses the shortcomings of traditional log‑collection agents, compares Flume and Filebeat based on low‑cost, stability, efficiency and lightweight criteria, and presents practical solutions for file discovery, offset tracking, multi‑line handling and performance tuning in modern logging pipelines.

Agent DesignFlumeObservability
0 likes · 13 min read
Design and Evaluation of Log Collection Agents: Flume vs Filebeat
21CTO
21CTO
Oct 30, 2020 · Big Data

Which Log Collection System Wins? Scribe, Chukwa, Kafka, Flume & ELK Compared

This article reviews the background, requirements, and architectural designs of major open‑source log collection systems—including Facebook’s Scribe, Apache’s Chukwa, LinkedIn’s Kafka, Cloudera’s Flume—and evaluates mature monitoring tools such as ELK, highlighting their features, use cases, advantages, and drawbacks for large‑scale log processing.

Big DataELKFlume
0 likes · 18 min read
Which Log Collection System Wins? Scribe, Chukwa, Kafka, Flume & ELK Compared
Efficient Ops
Efficient Ops
Aug 4, 2020 · Operations

Mastering Filebeat: How to Collect and Ship Container Logs to Kafka

This article introduces Filebeat as a lightweight log shipper, explains its core components and processing flow, and provides step‑by‑step configuration examples for gathering container logs and forwarding them to Kafka or Elasticsearch in cloud‑native environments.

ElasticsearchGoKafka
0 likes · 13 min read
Mastering Filebeat: How to Collect and Ship Container Logs to Kafka
dbaplus Community
dbaplus Community
Aug 3, 2020 · Operations

How iQIYI Built a Full‑Link Automated Monitoring Platform for Microservices

iQIYI’s tech product team designed a unified full‑link automated monitoring platform that integrates link, metric, and log collection with deep analysis, enhancing fault localization, performance insight, and scalability across microservices, while addressing limitations of existing tools like ELK, Prometheus, and Dapper.

MetricsMonitoringObservability
0 likes · 15 min read
How iQIYI Built a Full‑Link Automated Monitoring Platform for Microservices
Top Architect
Top Architect
Jun 28, 2020 · Operations

Design and Implementation of a Log Collection Agent: Challenges and Solutions

This article explains the evolution of logging, the role of log‑collection agents, industry solutions, and step‑by‑step techniques for building a reliable push‑mode log collector on Linux, covering file discovery, offset management, file identification, update detection, and safe resource release.

Agent DesignFile MonitoringLinux
0 likes · 17 min read
Design and Implementation of a Log Collection Agent: Challenges and Solutions
Zhuanzhuan Tech
Zhuanzhuan Tech
Feb 26, 2019 · Cloud Native

Automated Business Log Collection in Zhaozhuan Container Cloud Platform Using Log‑Pilot

This article describes how Zhaozhuan built an automated, business‑transparent log‑collection solution for its container cloud platform by evaluating several approaches, adopting Alibaba Cloud's open‑source log‑pilot, customizing its deployment, and addressing practical issues such as time‑zone bugs, latency, and duplicate collection.

Cloud NativeFluentdFlume
0 likes · 13 min read
Automated Business Log Collection in Zhaozhuan Container Cloud Platform Using Log‑Pilot
Meituan Technology Team
Meituan Technology Team
Oct 11, 2018 · Mobile Development

Logan: Meituan-Dianping Mobile Logging Framework Overview

Logan, Meituan‑Dianping’s open‑source mobile logging framework, unifies log collection, secure storage, backend processing, and visual frontend tools to capture all user‑side logs without sampling, enabling fast case‑analysis debugging across Android, iOS, mini‑programs and web, with future ML‑driven insights.

AndroidOpen SourceiOS
0 likes · 13 min read
Logan: Meituan-Dianping Mobile Logging Framework Overview
Ops Development Stories
Ops Development Stories
Oct 3, 2018 · Operations

Master Logstash: Essential Commands and Top Log Collection Plugins

This guide walks through Logstash fundamentals, from creating basic pipelines with input, filter, and output sections to using common plugins such as grok, mutate, date, geoip, multiline, and integrations with NGINX, rsyslog, Redis, and Docker‑based Logspout, providing practical configuration examples and command‑line tips.

DockerElasticsearchGrok
0 likes · 27 min read
Master Logstash: Essential Commands and Top Log Collection Plugins
21CTO
21CTO
Sep 2, 2018 · Operations

How We Built a Real‑Time Log Collection System with Go, Kafka, and Etcd

This article examines the shortcomings of a legacy log‑collection setup, introduces a streamlined real‑time architecture that eliminates per‑machine Hadoop clients, centralizes configuration via Etcd with a web UI, and details the Go implementation—including configuration design, Etcd watching, Kafka integration, tail management, and per‑second rate limiting—complete with code snippets and performance graphs.

EtcdReal-timelog collection
0 likes · 16 min read
How We Built a Real‑Time Log Collection System with Go, Kafka, and Etcd
Snowball Engineer Team
Snowball Engineer Team
Mar 23, 2018 · Big Data

Redesigning Snowball's Log Collection Architecture During Hadoop Cluster Expansion

The article details Snowball's challenges with a saturated CDH Hadoop cluster, outlines the limitations of the original Kafka‑based log pipeline, and explains how a comprehensive redesign using FlumeNG, Spillable Memory Channels, and custom HDFS sinks resolves latency, data loss, and high‑load issues while supporting future growth.

Cluster MigrationFlumeNGHadoop
0 likes · 6 min read
Redesigning Snowball's Log Collection Architecture During Hadoop Cluster Expansion