How iLogtail Evolved Over 13 Years to Lead Cloud‑Native Observability

iLogtail Development Overview

iLogtail is a pioneering lightweight log collector that has been evolving for 13 years, focusing on efficient extraction, processing, and transmission of observability data to Alibaba Cloud Log Service or other platforms. This article reviews its technical evolution and future direction.

History of Collection Agents

2013: First version launched with Feitian 5K system, aimed to unify logs from thousands of machines. Key features: inotify‑based real‑time collection, log parsing, remote storage, basic self‑monitoring.

2015: Alibaba’s cloud migration increased user base and demands, leading to high‑watermark feedback queues, checkpoint mechanisms, multi‑tenant management, and richer processing.

2017: With SLS commercialization and ACK service, user growth exploded; Go plugin system introduced, enabling container log collection, K8s metadata tagging, and entry into time‑series and tracing data.

2022: iLogtail open‑sourced 1.0.0, becoming a full‑stack observability collector with complete container runtime support and profiling data ingestion.

2024: Open‑source two‑year anniversary, release of 2.0.0 with significant improvements in usability, performance, and reliability.

Industry Milestones of Collection Agents

2011: Hadoop 1.0.0 released, marking big‑data maturity; Flume also open‑sourced.

2013: Logstash 1.0.0 released.

2014: Docker 1.0 launched, ushering the container era.

2015: Kubernetes 1.0.0 released, revolutionizing compute paradigms.

2016: Beats and Fluentd 1.0.0 released, integrating tightly with K8s.

2017: Prometheus 2.0 released, standardizing metric collection.

2017‑2018: Major cloud providers released commercial K8s services (AWS, Azure, GKE).

2019: FluentBit, Vector, and OpenTelemetry Collector all reached 1.0.0, each emphasizing performance, metric handling, and ecosystem integration.

Key Trends in Collection Agent Technology

High Performance : Modern agents (FluentBit, iLogtail, Vector) are written in C/C++/Rust for low‑cost, high‑throughput processing.

High Reliability : Memory limits, back‑pressure, disk buffering, and multi‑tenant isolation are now standard.

Flexibility : Modular, plugin‑based designs allow extensions in various languages; SPL, SQL, VRL, and OTTL provide expressive data processing.

Cloud‑Native Support : Tight integration with Kubernetes, CRDs, and Operators for automatic discovery and tagging.

All‑in‑One Capability : Agents now handle logs, metrics, traces, and profiling, reducing the need for multiple specialized tools.

Intelligence : Emerging AI‑driven features aim for auto‑configuration, auto‑discovery, and end‑to‑end automation.

From Logs to General Observability

Initially iLogtail only collected logs, focusing on real‑time, stable ingestion, handling log rotation, network anomalies, and basic self‑monitoring. With the rise of cloud‑native and K8s, iLogtail added container stdout collection and later introduced a Go plugin system (Input, Processor, Aggregator, Flusher) that enabled new data sources such as system‑journald, Windows events, and MySQL binlog.

The platform subsequently expanded to collect metrics, traces, and profiling data, supporting over 40 input plugins and 10 output plugins, and achieving full OpenTelemetry compatibility.

From Extensibility to Programmability

Early iLogtail’s monolithic pipeline limited extensibility. The Go plugin framework introduced modularity but added cross‑language overhead. In version 2.0, the C++ pipeline was also modularized, allowing independent plugins for input, processing, and flushing, and enabling seamless chaining between Go and C++ pipelines.

Memory management was optimized using Memory Arena and Zero‑Copy techniques, reducing allocations and improving multi‑line log splitting performance by up to 7×.

Core C++ functions such as filtering and masking were rewritten, delivering up to three‑fold speed gains for container stdout collection.

Configuration Structure Optimization

Legacy configuration files became deeply nested and inflexible. iLogtail 2.0 adopted a structured YAML format with clear sections for inputs, processors, and flushers, simplifying pipeline definition and enhancing extensibility.

Code Structure Refactoring

The codebase was split into a Pipeline core framework and Plugin modules, clarifying responsibilities and improving maintainability. New interfaces for Input, Processor, and Flusher promote community contributions.

Self‑Monitoring Framework Enhancement

Version 2.0 introduced a lock‑free, extensible self‑monitoring system where plugins can register custom metrics, leveraging atomic variables and read‑write separation for high performance.

Community Contributions and External Repository Integration

Developers have contributed plugins for Kafka, Elasticsearch, and Go 1.19 upgrades. An external‑plugin integration scheme using plugins.yml and external_plugins.yml enables private repositories to merge with the main codebase during pre‑compilation.

Conclusion and Outlook

iLogtail’s evolution mirrors the broader shift from host‑centric logging to cloud‑native, full‑stack observability. With continuous performance optimizations, modular architecture, and intelligent features, the project is poised to remain a leading open‑source collector, now rebranded as LoongCollector, driving the future of data collection.