Why iLogtail’s Open‑Source Cloud‑Native Agent Is Redefining Observability

Observability Discussion

In June 2022, Alibaba Cloud fully open‑sourced iLogtail, releasing a community edition that matches the enterprise version in kernel capabilities. iLogtail, the default collector for Alibaba Cloud SLS, has served millions of installations, ingesting tens of petabytes of observability data daily for monitoring, troubleshooting, operations analysis, and security use cases.

Observability in Everyday Life

Observability means inferring a system's internal state from its external outputs. Everyday examples include a car dashboard, which lets non‑engineers quickly gauge vehicle health, and medical diagnostics, where tools like ECG and X‑ray transform the human body from a black box into a transparent system, emphasizing the need for quantitative, data‑driven insight.

Opportunities and Challenges

Modern software development’s rapid iteration, distributed architectures, containerization, and cloud‑native environments increase system complexity, demanding observability data collectors that integrate tightly with DevOps pipelines, support dynamic, container‑based workloads, and handle diverse data sources.

Data Foundations of Observability

Logs, Traces, and Metrics form the three pillars of observability data, each with distinct characteristics and use cases. Logs capture detailed system events, Metrics provide lightweight statistical aggregates, and Traces represent structured call graphs. These data types can be transformed: structured logs can become Traces, which can be aggregated into Metrics.

Open‑Source Solutions

Typical open‑source observability stacks consist of five layers: collection agents (e.g., Filebeat, Fluentd, iLogtail), message queues (Kafka, RabbitMQ), processing engines (Flink, Logstash), storage/analysis engines (Elasticsearch, ClickHouse, Loki), and visualization tools (Kibana, Grafana). iLogtail, as the SLS‑official collector, inherits most enterprise features in its community edition.

iLogtail Evolution

Originating from Alibaba Cloud's ShenNong project in 2013, iLogtail evolved from basic file discovery to handling multi‑tenant isolation, supporting Docker and containerd, and adding Metric and Trace collection by 2021. In 2021 the Golang plugin code was open‑sourced, followed by the C++ core in 2022, enabling developers to build full cloud‑native observability pipelines.

Core Advantages of iLogtail

Lightweight

Designed with a C++ core and Golang plugins, iLogtail minimizes CPU and memory usage, crucial for large‑scale deployments where each node runs an agent.

High‑Efficiency Collection

iLogtail combines polling and inotify mechanisms, merging events to avoid duplication and automatically downgrading polling when inotify is available, achieving millisecond‑level latency and high throughput.

Ordered Log Collection

It uses dev+inode+signature (hash of the first 1 KB) to uniquely identify files across rotations, ensuring correct ordering even during rapid file turnover.

Reliability

iLogtail maintains open file descriptors for rotated logs, checkpoints progress to local files, and restores state after crashes or upgrades, preventing data loss while handling network spikes and process restarts.

Plugin Extensibility

type Processor interface {
    Init(Context) error
    Description() string
    ProcessLogs(logArray []*protocol.Log) []*protocol.Log
}

Developers can implement custom processors in Go, extending functionality without modifying the core.

Kubernetes Collection

iLogtail supports three deployment modes in K8s: DaemonSet (node‑wide collection), Sidecar (per‑pod isolation), and Deployment (centralized collection). It discovers containers via Docker/Containerd sockets, captures standard output and container file logs, enriches logs with pod and namespace metadata, and offers CRD‑based configuration (enterprise only).

Multi‑Tenant Isolation

A time‑slice scheduler ensures fair resource allocation across configurations, while multi‑level high/low water‑mark queues provide back‑pressure control between reading, parsing, and sending stages.

Enterprise vs. Community Editions

Both editions share the same core collection capabilities. The enterprise version adds seamless integration with Alibaba Cloud SLS, richer OS and architecture support (including Windows and ARM), automated deployment via Alibaba Cloud services, and premium support.

Cloud‑Native Observability Platform on SLS

SLS offers unified storage and analysis for Logs, Metrics, and Traces, enabling rapid construction of observability platforms for IT‑Ops, Dev‑Ops, Sec‑Ops, and Business‑Ops workloads.

Open‑Source Community Outlook

The iLogtail project encourages community contributions, plans to open‑source a global control plane, and aims to adopt OpenTelemetry standards and an iLogtail Operator for native Kubernetes experiences.