Alibaba Cloud SLS provides 8 Agent Skills covering the full log lifecycle — collection, indexing, query, alerting, visualization, and root cause analysis — enabling natural language-driven log management with concrete installation commands and practical scenarios for monitoring and troubleshooting.
A rigorous benchmark of nine Kubernetes log collectors under a 1‑core, 1 GiB limit shows VictoriaMetrics' vlagent achieving 143,000 lines per second—4.5× faster than Fluent Bit and 28× faster than Fluentd—while using the least CPU and memory, and exposing hidden correctness bugs in several competitors.
This guide walks through a systematic, evidence‑driven process for diagnosing and lowering Filebeat’s excessive CPU, memory, and file‑handle consumption—including version checks, process sampling, retry and queue analysis, registry handling, input configuration, parsing rules, queue sizing, output tuning, monitoring, and safe rollback procedures.
This guide walks through selecting Filebeat OSS 7.10.2, preparing Ubuntu 20.04, uploading and extracting the package, configuring filebeat.yml for Easysearch log paths, creating an index template, starting Filebeat, verifying data ingestion, and applying production‑grade recommendations such as systemd service setup and ILM policies.
LoongCollector introduces a One‑Time file collection mode that scans matching files once, records a snapshot, and exits, enabling efficient historic log migration, data back‑fill, and temporary batch processing while providing fine‑grained checkpoints, execution windows, and throttling controls to avoid quota issues and ensure reliable completion.
LoongCollector’s OneTime file collection feature enables fast, reliable migration of historical logs, data back‑filling, and batch processing by scanning files once, using checkpoints for fault tolerance, configurable execution windows, and rate‑limiting to avoid impacting live data streams.
This article explains why the decades‑old Syslog protocol remains relevant in modern microservice and cloud‑native environments, detailing its structure, advantages, typical use cases, step‑by‑step server setup with rsyslog, command‑line analysis techniques, integration with Java applications, and a comparison with ELK/Loki solutions.
This guide explains what Filebeat is, how its pre‑built modules work, and provides step‑by‑step instructions for installing Filebeat and configuring it to output logs to files, Logstash, Elasticsearch, Redis, and multiple destinations within an ELK stack.
This guide explains why modern enterprises need unified log collection for diverse security devices, introduces LoongCollector's flexible configuration for parsing and standardizing firewall logs from WAF, FortiGate, and Palo Alto, and provides step‑by‑step examples, code snippets, and visual results to enable efficient threat analysis and compliance.
This guide explains how to unify and parse heterogeneous firewall logs from vendors such as Longteng WAF, FortiGate, and Palo Alto by configuring LoongCollector, creating Alibaba Cloud Log Service resources, and applying JSON, key‑value, and CEF parsing rules to enable structured security analytics.
The article explores how automated thread dumps can be expanded into multiple testing scenarios—including log collection, resource monitoring, fault injection, performance result analysis, and environment cleanup—by leveraging Kubernetes APIs, Prometheus, Chaos Mesh, and scripting tools to improve efficiency, observability, and system resilience.
LoongCollector, the 2025 evolution of iLogtail, introduces a fully redesigned pipeline architecture, hot‑reload isolation, significant CPU and memory reductions, and advanced monitoring, delivering up to 80% higher log‑collection throughput for cloud‑native environments while ensuring seamless upgrades and multi‑region fault tolerance.
In a series of controlled benchmarks on an Alibaba Cloud ECS instance, the next‑generation LoongCollector log‑collection agent demonstrated dramatically higher throughput, lower CPU usage, and more efficient memory consumption than its predecessor iLogtail and popular open‑source alternatives such as FluentBit, Vector, and Filebeat across multiple log formats and traffic levels.
LoongCollector, the 2025 evolution of iLogtail, delivers a comprehensive overhaul of log collection with high‑performance C++/Go pipelines, dynamic plugin combos, hot‑reload isolation, adaptive network throttling, enhanced tag handling, seamless migration, and robust monitoring for cloud‑native environments, achieving up to 80% higher throughput while cutting CPU usage by 35% and memory by 10%.
This article explains how enterprises can overcome the challenges of globally collecting logs by using Alibaba Cloud's high‑performance iLogtail/LoongCollector agents, selecting optimal network paths, applying cost‑saving strategies such as CloudLens diagnostics and LZ4 compression, and configuring multi‑region dual‑write for reliable, low‑cost log delivery.
This guide explains how enterprises can reliably and cost‑effectively gather logs from distributed overseas environments into Alibaba Cloud Log Service (SLS) by selecting the right network path, leveraging iLogtail/LoongCollector agents, applying compression, filtering, multi‑region routing, and using CloudLens for monitoring and cost control.
This article details a real‑world case where massive multi‑line log volumes overwhelmed iLogtail, explains the performance bottlenecks caused by full‑line regex matching, describes the switch to prefix‑only matching and IngestProcessor, and shows how these changes lifted throughput from under 100 MB/s to over 300 MB/s while halving CPU usage.
Didi replaced its legacy agent with Alibaba’s open‑source iLogTail, re‑architected it to use a shared thread‑pool and SIMD‑accelerated parsing, rewrote critical plugins in C++ and added robust Kafka retry logic, achieving over twice the throughput while cutting CPU usage by more than half and maintaining near‑zero latency at massive scale.
This article explores eight practical scenarios for using message queues—such as asynchronous processing, service decoupling, traffic shaping, delayed tasks, log collection, distributed transactions, remote calls, and broadcast notifications—providing code examples with RabbitMQ, RocketMQ, and Kafka to illustrate each pattern.
This step‑by‑step guide shows how to use a Helm chart to deploy Filebeat in a Kubernetes cluster, automatically collect container logs, and forward them to a Kafka cluster for reliable, scalable observability.
This guide explains how to install Filebeat, configure its built‑in modules, and set up outputs to files, Logstash, Elasticsearch, Redis, and multiple destinations, providing a lightweight, Java‑free solution for centralized log collection within an ELK architecture.
This article investigates why enabling line‑prefix regex for multiline logs slows Logtail down, explains the underlying regex matching mechanism, and demonstrates how switching from boost::regex_match to boost::regex_search with proper flags can dramatically improve collection throughput, achieving a seven‑fold speed increase.
Celebrating iLogtail’s two‑year open‑source anniversary, this article explores the journeys of community committers, their technical contributions such as Kafka dynamic‑topic support and data flattening, and how their work advances cloud‑native observability and log collection for enterprises.
This article explains how to use Filebeat as a lightweight log shipper together with Graylog for centralized log aggregation, storage, and analysis, covering installation, configuration files, Docker deployment, Spring Boot integration, and query techniques for efficient operations monitoring.
This article examines the traditional ELK stack, compares iLogtail with Filebeat and Logstash in real‑world performance tests, analyzes why iLogtail could not previously replace Logstash, and presents five concrete engineering solutions that enable iLogtail to become a viable, high‑performance alternative for log collection and processing.
iLogtail, a lightweight log collector, has transformed over 13 years from a simple log‑gathering tool into a full‑stack, cloud‑native observability platform, introducing Go plugins, high‑performance C++ pipelines, SPL processing, modular architecture, and advanced self‑monitoring, reflecting broader trends in data collection technology.
This article chronicles iLogtail's 13‑year journey—from its 2013 inception as a basic log collector to a fully open‑source, cloud‑native observability platform—highlighting technical milestones, emerging trends in log agents, architectural innovations, performance breakthroughs, and future directions.
This article explains Kubernetes container runtimes, the CRI and OCI standards, compares Docker, containerd, CRI‑O and runc, outlines the challenges of log collection in K8s, and shows how open‑source collectors like Filebeat and Fluent Bit as well as iLogtail's daemonset and sidecar modes provide flexible, enriched logging solutions.
A two‑year community survey of the open‑source iLogtail collector reveals that high performance, container‑friendly design, extensive plugin ecosystem, and strong Kubernetes integration drive widespread production use, while users request better documentation, a more polished ConfigServer tool, and clearer contribution pathways.
This article provides a comprehensive guide on building a unified log collection system using Filebeat and Graylog, covering the tools' concepts, configuration files, component functions, Docker deployment, and practical code examples for efficient log monitoring across multiple environments.
This article provides a comprehensive, lightweight solution for collecting Kubernetes logs using Grafana Loki, covering its advantages, component comparison, deployment modes (All‑In‑One, microservices, bare‑metal), required configuration files, ConfigMap and PersistentVolume setup, Promtail installation, Helm deployment, and common troubleshooting steps.
iLogtail 2.0 replaces the monolithic, file‑oriented design of its predecessor with a modular pipeline configuration, new input/processor/output plugins, a refreshed API, SPL processing, finer‑grained parsing controls, nanosecond‑level timestamps, enhanced observability, and performance improvements, while providing compatibility guidance for both commercial and open‑source editions.
This article explains the sidecar concept, compares it with SDK approaches, and provides detailed Kubernetes examples—including a log‑collection sidecar, a request‑forwarding sidecar, and an HTTP‑intercepting sidecar—complete with YAML manifests and Rust and Scala code to demonstrate implementation and deployment.
This article details the end‑to‑end design of Vivo’s custom log‑collection agent, covering file discovery with inotify, unique file identification using inode and content hash, real‑time reading via RandomAccessFile, checkpointing, Kafka integration, offline HDFS ingestion, resource throttling, and platform‑wide management, while comparing it with open‑source alternatives.
This article explains how to use Filebeat together with Graylog to collect, ship, store, and analyze logs from multiple environments, covering tool introductions, configuration files, Docker deployment, Spring Boot integration, and practical search syntax for effective log monitoring.
Facing data growth, reliability demands, and high maintenance costs, a company redesigned its logging stack by replacing ELK with a Kubernetes‑native pipeline built on Log‑Pilot, Vector, and ClickHouse, achieving lower cost, higher performance, and seamless migration while preserving familiar query interfaces.
This article explains the concepts and importance of observability, introduces the Vector data‑pipeline tool and its architecture, demonstrates how to configure sources, transforms and sinks, and shows how to integrate Vector with the Honghu platform to build a complete, real‑time monitoring solution for modern distributed systems.
This article compares four popular open‑source log collection tools—Logstash, Fluentd, Fluent Bit, and Vector—examining their key features, performance, resource usage, scalability, security, and ecosystem to help enterprises select the most suitable solution for their specific logging needs.
This article compares traditional host‑based log collection with Kubernetes container logging, outlines the challenges of dynamic pod migration, diverse storage methods, and metadata injection, and evaluates deployment options such as DaemonSet, Sidecar, and various volume types for reliable log aggregation.
Categraf is an all‑in‑one, Go‑based monitoring agent that consolidates metric, log, and trace collection, offering remote_write support, lightweight deployment, and extensive plugin configurations to replace multiple exporters in Prometheus‑based observability stacks.
This article examines the challenges of scaling log storage and retrieval for high‑traffic services, analyzes the cost and performance limits of traditional ELK‑based pipelines, and presents a streamlined, UDP‑driven architecture using ClickHouse that dramatically reduces hardware expenses while handling hundreds of gigabytes per second.
This article explains how container log collection in Kubernetes differs from traditional host logging, outlines common deployment methods such as DaemonSet and Sidecar, compares log storage options, and offers practical guidance on handling stdout and file‑based logs for reliable operations.
This article traces Zhuanzhuan's evolution of log collection—from a bare‑metal scribe + flume pipeline, through a container‑aware log‑pilot solution, to a cloud‑native filebeat and fb‑advisor architecture—detailing the motivations, technical designs, performance gains, and trade‑offs of each stage.
This article traces ZuanZuan's log‑collection journey from the bare‑metal era through containerization to a cloud‑native solution, detailing the challenges, custom developments like log‑pilot, ByteCompass, and fb‑advisor, and the performance gains achieved with each transition.
This article explains the types and value of observability data, the characteristics and data‑collection requirements of Kubernetes deployments, the challenges of large‑scale log, metric and trace ingestion, and practical solutions such as DaemonSet vs Sidecar, ConfigServer, CRD automation, serverless support, and eBPF integration.
Loggie is a cloud-native, Go-based log-collection agent that replaces Filebeat and Flume by using a micro-kernel producer-consumer architecture with hot-swappable pipelines, achieving 2 GB/s read speeds, 1.6‑2.6× higher throughput while using only a quarter of the CPU, and providing built-in observability, reliability, and latency monitoring for large-scale enterprise deployments.
This article presents the design principles, core techniques, and practical solutions of Vivo's self‑developed Bees log collection agent, covering file discovery, unique identification, real‑time and offline ingestion, checkpointing, resource control, platform management, and a comparison with open‑source alternatives.
This article provides an extensive overview of big‑data interview subjects, covering browser and mobile log collection methods, data synchronization techniques (batch, real‑time, sharding), offline data development platforms, streaming architectures, data service evolution, performance optimization, and data‑mining layers and applications.
Vivo’s Bees‑agent is a custom, lightweight log‑collection service that discovers rotating files via inotify, uniquely identifies them with inode and hash signatures, supports real‑time and offline ingestion to Kafka and HDFS, offers checkpoint‑resume, resource isolation, rich metrics, and a centralized management platform, outperforming open‑source collectors in latency, memory usage, and scalability.
NetEase Yanxuan built a unified log platform that consolidates collection, delivery, processing, storage, analysis, and alerting for near‑real‑time and offline logs, using Flume, a Golang‑based Loggie agent, Kafka, Flink, HBase and Elasticsearch to ensure high performance, data‑quality, container‑native isolation, full‑link traceability, and automated scaling, with Loggie now open‑sourced.
The article introduces the ZuanZuan cloud platform, detailing its overall architecture, image management workflow that abstracts Dockerfiles, release‑upgrade strategies with custom controllers, container monitoring evolution to Prometheus, and log‑collection mechanisms that handle large Java‑based log volumes.
This article explores the open‑source release of Alibaba Cloud's iLogtail, detailing its lightweight, high‑performance design, multi‑tenant isolation, plugin architecture, Kubernetes integration, and the differences between its enterprise and community editions, while highlighting its role in modern observability pipelines.
This guide compares three log‑collection methods—SLS, NAS, and Kafka—available in Alibaba Cloud Serverless App Engine, explaining their architectures, ideal use cases, configuration details, and how to avoid common pitfalls such as log loss and rotation issues.
This article presents a comprehensive overview of NetEase's log collection and transmission platform, detailing its evolution from 2011 to the current Datastream‑NG architecture, the system's design goals, core component optimizations, operational monitoring, and future plans for intelligent scaling and diagnostics.
This guide explains how to install Fluentd via Docker or manual methods, configure the environment (NTP, file descriptors, kernel parameters), use the td-agent package, manage services with systemd, send test logs, and understand Fluentd's event lifecycle, filters, labels, and configuration directives.
The article analyses the massive cost and performance challenges of traditional log‑storage pipelines at JD.com, then proposes a streamlined architecture that eliminates disk and MQ stages, uses in‑memory buffering, UDP transport, Snappy/ZSTD compression, and ClickHouse storage to achieve multi‑gigabyte per‑second throughput with over 70% hardware cost reduction.
This article introduces the Filebeat log‑shipping tool and the Graylog log‑management platform, explains their architectures and workflows, provides detailed configuration examples for Filebeat inputs, modules, and Graylog components, and walks through deployment using Docker, Docker‑Compose, and native package installations.
This article recounts NetEase's journey from early host‑based log collection to operator‑driven Kubernetes logging, discusses the challenges of large‑scale log ingestion, evaluates existing agents, and introduces the open‑source Loggie project with its architecture, features, performance gains, and roadmap.
This article explains how Filebeat’s harvester and prospector components read and forward system logs, maintain file offsets in a registry, and provides a sample YAML configuration for collecting logs from a specified file and sending them to Elasticsearch, illustrating key operational concepts for log management.
This article explains how to use Filebeat as a lightweight log shipper and Graylog as a centralized logging platform, covering their architectures, configuration files, Docker deployment steps, and UI features to help engineers efficiently collect, process, and analyze logs across multiple environments.
This guide demonstrates multiple ways to gather Nginx access and error logs—directly with Filebeat to Elasticsearch, via Filebeat to Logstash then Elasticsearch, and using rsyslog to forward logs to Logstash—providing step‑by‑step configurations, code snippets, and visual illustrations for each method.
This article describes the background, architecture, and implementation of a centralized database log collection and analysis platform built in 2021, detailing how logs from hosts, containers, and databases are normalized, streamed through Kafka, processed with Flink, stored in Elasticsearch, visualized with Kibana, and extended with alerting and configuration management to improve fault diagnosis and lay the groundwork for future AI‑driven operations.
This guide shows how to configure Fluent Bit to collect Kubernetes container logs, develop a Go‑based ClickHouse output plugin, deploy the pipeline with ConfigMap, RBAC and DaemonSet resources, define the ClickHouse table schema, and evaluate performance across different log rates.
This article explains how to deploy and configure Filebeat and Graylog for centralized log collection, covering installation methods, configuration files, Docker deployment, input modules, pipelines, and practical examples for efficiently gathering and analyzing logs across multiple environments.
This article analyzes the challenges of massive log storage and retrieval, calculates the bandwidth and hardware costs of traditional pipelines, and presents a streamlined architecture that uses in‑memory buffering, UDP transport, compression, and ClickHouse to achieve petabyte‑scale throughput while cutting storage costs by over 75%.
This article details the open‑source release of Alibaba’s iLogtail, explains the observability challenges it solves, traces its evolution through the Feitian 5K, Alibaba Group, and cloud‑native stages, highlights its high‑performance, stability, and multi‑tenant features, and outlines the community‑driven open‑source goals alongside a brief Impala overview.
This article explains how to use Filebeat together with Graylog to collect, process, and visualize logs from multiple services and environments, covering tool introductions, configuration files, component details, deployment methods, and practical code examples.
This guide explains how to use Filebeat to ship logs to Graylog, covering Filebeat's architecture, configuration files, deployment options with Docker or native packages, Graylog's components and pipelines, and step‑by‑step Docker‑compose setup for a scalable centralized logging solution.
The article analyzes the high cost and scalability challenges of traditional log collection pipelines and proposes a streamlined architecture that uses in‑memory buffering, UDP transport, aggressive compression, and ClickHouse storage to achieve massive throughput while drastically reducing hardware and operational expenses.
This article introduces a lightweight, cross‑platform SDK for mini‑programs that unifies log collection—including exception, normal, and performance data—by intercepting App, Page and platform‑specific APIs, handling multi‑environment differences, and providing flexible usage via npm or single‑file integration.
Learn step‑by‑step how to gather Nginx access and error logs using Filebeat, Logstash, and rsyslog, configure each component to forward logs to Elasticsearch, visualize them in Kibana, and choose the most suitable pipeline for flexible log preprocessing and storage.
This guide walks through three practical methods for harvesting Nginx access and error logs—directly with Filebeat to Elasticsearch, via Filebeat‑Logstash‑Elasticsearch pipeline, and using Rsyslog to forward logs to Logstash—complete with configuration snippets and visual illustrations.
This article outlines how to design, print, collect, and manage online service logs efficiently—covering log levels, key information, formatting, rolling, local vs. remote storage, real‑time collection, and tool selection—to turn logs into a valuable debugging and analytics asset.
This article walks through redesigning a Go‑based log‑collection framework, introducing etcd for distributed configuration, demonstrating context for timeout and data propagation, and showing how to integrate Kafka consumers while improving concurrency handling and adding rate‑limiting mechanisms.
This article shows how to use the Unix tail command together with a simple Java program to collect live logs, explains the crucial difference between tail -f and tail -F during file rotation, and provides practical steps and commands for reliable log monitoring.
This guide demonstrates multiple methods for gathering Nginx access and error logs—directly with Filebeat to Elasticsearch, via Logstash for preprocessing, and through rsyslog forwarding—detailing configuration steps, required code snippets, and verification using Kibana and Elasticsearch-head.
This tutorial demonstrates how to gather Nginx access and error logs using Filebeat, Logstash, and Rsyslog, configure each component, and ship the logs to an Elasticsearch cluster for visualization in Kibana, offering three practical collection methods.
This article explains the challenges of container log collection in Kubernetes, evaluates log‑agent options, details the design of a custom Filebeat‑based controller architecture, shares performance tuning with Golang, and outlines monitoring and future extensions for a robust cloud‑native logging solution.
This article provides a detailed tutorial on Filebeat 7.7.0, covering its purpose, architecture, installation via tarball, essential commands, configuration of inputs and outputs (including Logstash and Elasticsearch), keystore usage, module activation, and step‑by‑step verification of log collection in an ELK stack.
This article explains how NetEase’s Lightboat micro‑service platform tackles the challenges of log collection in cloud‑native Kubernetes environments by designing a custom controller, selecting Filebeat as the agent, integrating it via DaemonSet, extending its functionality, and applying Golang performance‑tuning techniques to achieve efficient, observable logging.
This article walks through designing and implementing a Go log‑collection agent that uses etcd for configuration storage, context for timeout and metadata handling, and Kafka for message consumption, complete with code examples, setup instructions, and a rate‑limiting utility.
This article shows how to use the Linux tail command together with a simple Java program to collect logs in real time, explains the difference between tail -f and tail -F, demonstrates file rotation handling, and offers practical tips for reliable log monitoring.
This article analyses the shortcomings of traditional log‑collection agents, compares Flume and Filebeat based on low‑cost, stability, efficiency and lightweight criteria, and presents practical solutions for file discovery, offset tracking, multi‑line handling and performance tuning in modern logging pipelines.
This article explains how to design a scalable log collection system for big‑data platforms by combining Flume for data ingestion, Kafka for buffering and high‑throughput transport, and downstream processing components, providing configuration examples and best‑practice recommendations.
This article reviews the background, requirements, and architectural designs of major open‑source log collection systems—including Facebook’s Scribe, Apache’s Chukwa, LinkedIn’s Kafka, Cloudera’s Flume—and evaluates mature monitoring tools such as ELK, highlighting their features, use cases, advantages, and drawbacks for large‑scale log processing.
This article introduces Filebeat as a lightweight log shipper, explains its core components and processing flow, and provides step‑by‑step configuration examples for gathering container logs and forwarding them to Kafka or Elasticsearch in cloud‑native environments.
iQIYI’s tech product team designed a unified full‑link automated monitoring platform that integrates link, metric, and log collection with deep analysis, enhancing fault localization, performance insight, and scalability across microservices, while addressing limitations of existing tools like ELK, Prometheus, and Dapper.
The article describes the challenges of troubleshooting distributed deployments across many machines and presents a solution built on the ELK stack that centralizes logs from Java and Go services, enabling near‑real‑time search, visualization, and faster issue resolution.
This article explains the evolution of logging, the role of log‑collection agents, industry solutions, and step‑by‑step techniques for building a reliable push‑mode log collector on Linux, covering file discovery, offset management, file identification, update detection, and safe resource release.
This article introduces Filebeat as a container log collector, explains why it was chosen, outlines its core architecture and processing flow, and provides a practical configuration example for sending logs to Kafka, offering a solid foundation for further development and deeper source‑code analysis.
This guide explains how to gather Nginx access and error logs using Filebeat, Logstash, and rsyslog, and forward them to an Elasticsearch cluster for analysis, covering three practical pipelines with configuration examples and deployment steps.
This guide explains three practical methods for gathering Nginx access and error logs—directly with Filebeat to Elasticsearch, via Filebeat to Logstash then Elasticsearch, and using rsyslog to forward logs to Logstash—complete with configuration snippets and visualization steps in Kibana.
This article explains why log collection in Kubernetes is more complex than on traditional VMs, compares passive and active collection methods, evaluates stdout versus file output, and provides practical recommendations and deployment guides for small, medium, and large clusters.
Learn how to set up agent‑less log collection for Nginx using Rsyslog, forward logs via the omkafka module to a Kafka cluster, and process them with Logstash into Elasticsearch for visualization in Kibana, including installation, configuration, and testing steps.
This article describes how Zhaozhuan built an automated, business‑transparent log‑collection solution for its container cloud platform by evaluating several approaches, adopting Alibaba Cloud's open‑source log‑pilot, customizing its deployment, and addressing practical issues such as time‑zone bugs, latency, and duplicate collection.
Logan, Meituan‑Dianping’s open‑source mobile logging framework, unifies log collection, secure storage, backend processing, and visual frontend tools to capture all user‑side logs without sampling, enabling fast case‑analysis debugging across Android, iOS, mini‑programs and web, with future ML‑driven insights.
This guide walks through Logstash fundamentals, from creating basic pipelines with input, filter, and output sections to using common plugins such as grok, mutate, date, geoip, multiline, and integrations with NGINX, rsyslog, Redis, and Docker‑based Logspout, providing practical configuration examples and command‑line tips.