How iLogtail Achieves Million‑Scale Observability with SRE Practices

Background and Motivation

Rapid market saturation, increasing competition in public cloud, and frequent large‑scale outages have pushed enterprises to seek cost‑effective, high‑quality reliability engineering solutions. Transitioning from consumer‑focused (ToC) to business‑focused (ToB) models raises quality expectations, making stable, observable agents essential.

iLogtail Overview

iLogtail is Alibaba Cloud's self‑developed observability data collection agent, lightweight, high‑performance, and auto‑configurable, deployed on physical machines, VMs, and Kubernetes. It powers observability for major Alibaba services (Taobao, Tmall, Alipay, etc.) with millions of installations and tens of petabytes of data daily.

Stability Challenges Specific to iLogtail

Fast‑changing business and environments : new data sources, outputs, processing methods, and deployment contexts.

Weak client control : version convergence difficulty, diverse client environments, uncontrolled operator actions.

Massive deployment scale : millions of instances, single‑version releases affecting all agents.

Reliability Goals for the Agent

Data integrity : accurate collection even during short‑term upstream/downstream failures.

Reliability : continuous operation under high load, container restarts, crashes, or malformed input.

Performance impact : minimal CPU, memory, disk, and network usage.

Real‑time processing : low latency for near‑real‑time analytics, with no performance regression.

Reliability Engineering Elements and Methodology

The team emphasizes four pillars: awareness, standards, technology, and mechanisms.

Awareness

Embedding a quality‑first mindset across design, development, testing, and deployment ensures that reliability is a shared responsibility.

Standards

Adopting industry‑proven coding conventions (Google C++ style, Effective Go) and documenting design decisions reduces knowledge gaps.

Technology

Automation tools (CI/CD pipelines, static analysis, automated testing, chaos engineering) enforce standards and accelerate feedback.

Mechanisms

Integrating performance metrics, SLA/SLO/SLI definitions, and incentive structures aligns team behavior with reliability outcomes.

Design Phase: Plugin‑Based and Programmable Architecture

iLogtail uses a pipeline of lightweight processing plugins (JSON parsing, field replacement, etc.) and a custom SPL language for runtime‑compiled custom plugins, enabling flexible, high‑performance data handling without frequent code changes.

Development Phase

Two parallel release models support both open‑source (agile feature development) and commercial (stability‑first) versions. Branch‑by‑feature and trunk‑based development are combined to isolate large features while keeping the mainline stable. Code formatting containers (clang‑format, gofmt) enforce consistent style.

Testing Phase

Testing addresses parameter explosion, uncontrolled inputs, upstream/downstream dependencies, and diverse runtime environments.

Static analysis : tools like coverity and golangci‑lint catch memory leaks and unsafe calls early.

Unit tests : orthogonal array testing ensures comprehensive parameter coverage.

Functional & integration tests : custom E2E framework simulates host, container, and cloud environments, using Docker Compose, Alibaba Cloud ECS/ACK, and ChaosBlade for fault injection.

Performance tests : verify that new features meet latency and resource‑usage targets without regressions.

Precise Testing Strategy

Instead of exhaustive version testing, the team selects representative major versions and adds the first incompatible minor version identified via compatibility checks, dramatically reducing test matrix size while preserving coverage.

Release Phase: Gray‑Release Practices

Gradual rollouts are performed across clusters, users, regions, and user tiers. Metrics (resource usage, restart frequency, data volume) are monitored for at least 24 hours before expanding the release.

Operation Phase: SLA‑Driven Monitoring

Service Level Agreements (e.g., 99.9% availability) are backed by a small set of Service Level Objectives and Indicators, spanning system, service, business, data, and resource layers. The iLogtail team builds an intelligent observability platform on top of Alibaba Cloud SLS to aggregate logs, traces, and metrics for proactive issue detection.

Customer Support and Self‑Service Tools

Two self‑service tools—Cloud Lens for SLS (global view of agent instances, error aggregation, and remediation guidance) and Container Metadata Preview (visual mapping of container selection)—reduce ticket volume by up to 30%.

UI Improvements

Redesigning the configuration UI groups related parameters, collapses rarely used options, and clarifies dependencies, resulting in a 30% increase in onboarding efficiency and an 80% reduction in heartbeat‑related tickets.

Summary and Outlook

By embedding reliability engineering throughout the development lifecycle, iLogtail achieved an 80% increase in issue interception and eliminated OS‑compatibility problems. Future work will leverage large‑language models for automated root‑cause analysis, data‑ingestion automation, and further full‑stack SRE capabilities.