How iLogtail Evolved: From Simple Log Collector to Cloud‑Native Observability Platform

iLogtail Development Overview

iLogtail began in 2013 as a lightweight log collector for Alibaba's Feitian 5K distributed system, focusing on real‑time log ingestion via inotify, log parsing, and remote storage delivery. Early versions emphasized basic log collection and self‑monitoring.

By 2015, as Alibaba migrated core services to the cloud, iLogtail expanded to support higher throughput, multi‑tenant isolation, checkpoint mechanisms, and richer log processing to meet enterprise reliability demands.

In 2017, with the commercial launch of SLS and ACK, iLogtail added a Go plugin system, enabling container log collection, Kubernetes metadata tagging, and initial support for time‑series and tracing data.

2022 marked the full open‑source release of iLogtail 1.0.0, delivering complete container runtime support, ecosystem integrations, and profiling data collection, signaling its transition from a single‑purpose collector to a comprehensive observability data collector.

2024 saw the release of iLogtail 2.0.0, improving usability, performance, and reliability over the 1.x series.

Industry Context and Trends

The evolution of iLogtail mirrors broader cloud‑native developments: Hadoop (2011), Flume (2011), Logstash (2013), Docker (2014), Kubernetes (2015), Beats/Fluentd (2016), Prometheus (2017), and the rise of FluentBit, Vector, and OpenTelemetry Collector (2019) all set the stage for modern observability agents.

Key Technical Trends in Log Agents

High Performance – Modern agents (FluentBit, iLogtail, Vector) are written in C/C++/Rust for low‑overhead processing.

High Reliability – Features like memory limits, back‑pressure, disk buffering, and multi‑tenant isolation ensure data durability.

Flexibility – Modular plugin architectures (C++/Go plugins, WebAssembly) and diverse processing languages (SQL, SPL, VRL, OTTL) enable extensibility.

Cloud‑Native Support – Tight integration with Kubernetes via service discovery, metadata tagging, CRDs, and operators.

All‑In‑One Capability – Agents now collect logs, metrics, traces, and profiling data, often incorporating eBPF and instrumentation.

Intelligence – Emerging “auto‑config”, “auto‑discovery”, and end‑to‑end automation promise smarter data collection.

Architectural Evolution

The Go plugin pipeline consists of Input, Processor, Aggregator, and Flusher stages, allowing both Go and C++ components to interoperate. This enabled rapid addition of sources such as system‑journald, Windows Event, and MySQL binlog.

iLogtail introduced a unified observable data model (PipelineEvent) replacing the log‑centric protobuf, improving efficiency for metrics, traces, and profiling.

Both the Go and C++ pipelines were modularized in version 2.0, supporting flexible plugin composition, cross‑language chaining, and zero‑copy memory handling.

Programmable Data Processing (SPL)

SPL provides a concise, SQL‑like language for extracting, parsing, and filtering data, reducing the need for multiple plugins and improving developer productivity.

Performance Optimizations

iLogtail employs a single‑threaded event‑driven model, time‑slice scheduling, lock‑free structures, and zero‑copy techniques, achieving superior ingestion rates and low resource usage compared to peers.

Version 2.0 further optimizes memory via arena allocation, enhances multi‑line log splitting with a state‑machine redesign, and rewrites core C++ components (filtering, masking) for up to three‑fold speed gains.

Configuration and Code Structure Improvements

Configuration migrated from ad‑hoc formats to structured YAML with clear Input/Processor/Flusher sections, enabling easy extension and composability.

The codebase was refactored into a Pipeline core and Plugin modules, clarifying interfaces and simplifying contributions.

Self‑Monitoring Enhancements

A new lock‑free, extensible self‑monitoring framework allows plugins to register custom metrics, improving observability of the agent itself.

Community and Ecosystem Integration

Community contributors have added Kafka, Elasticsearch, and Go 1.19 support, among others. An external repository integration scheme using plugins.yml and external_plugins.yml enables private plugins to be merged at build time (see https://github.com/alibaba/ilogtail/pull/1708).

Future Outlook

iLogtail is rebranded as LoongCollector, aiming to continue evolving with emerging technologies to stay ahead in the observability space.