LoongCollector: 10× Throughput and 80% Resource Savings – In‑Depth Technical Analysis

LoongCollector, originally iLogtail, has evolved over a decade into a cloud‑native unified collector used by Alibaba Cloud, Ant Group and tens of thousands of enterprises, handling daily data volumes of hundreds of petabytes.

Performance Goals

The system targets three core metrics: high throughput (ability to sustain peak loads without backlog), low resource cost (CPU, memory, I/O) and stability (zero data loss under congestion).

Benchmark Methodology

Tests were run on an Alibaba Cloud ECS g7 instance (32 vCPU, 64 GB RAM, ESSD PL3 1500 GiB, Ubuntu 20.04). The full benchmark script is available on GitHub (benchmark e2e README).

Maximum Throughput : LoongCollector achieved 546 MB/s for single‑line logs, far exceeding FluentBit (36 MB/s), Vector (38 MB/s) and Filebeat (9 MB/s). For multi‑line logs, it reached 238 MB/s versus 24 MB/s (Vector) and 22 MB/s (FluentBit).

Resource Efficiency : At 10 MB/s input, LoongCollector used only 3.40 % CPU and 29.01 MB RAM, while FluentBit consumed 12.29 % CPU (+261 %) and 46.84 MB RAM (+61 %). Vector and Filebeat showed CPU usage up to 35.80 % (+952 %) and 83.24 % (+186 %) respectively.

Serialization Overhead : LoongCollector’s Protobuf wire‑format serialization reduced CPU for serialization from 12.5 % (others) to 5.8 % (‑54 %) and memory copies from three allocations to a single zero‑copy operation (‑67 %).

Key Architectural Innovations

Zero‑Copy String Handling : Uses a shared PipelineEventGroup with a SourceBuffer and string_view to store raw log data once, eliminating the typical four string copies in traditional agents.

Lock‑Free Event Pools : Thread‑local lock‑free pools provide direct reuse; a double‑buffer pool enables cross‑thread batch reclamation, cutting lock contention by ~90 % and reducing memory fragmentation by ~80 %.

Zero‑Copy Serialization : Directly writes Protobuf wire format without intermediate objects, avoiding extra allocations and copies.

Multi‑Tenant Isolation : Each pipeline has an independent bounded queue with high/low watermark feedback, providing back‑pressure, resource isolation and automatic recovery.

Priority‑Aware Scheduling : Guarantees strict priority handling and fair round‑robin within the same priority level, preventing starvation.

Adaptive Concurrency Limiter : An AIMD‑inspired limiter reduces concurrency rapidly when failure rates rise and ramps up slowly on recovery, protecting other destinations from cascading failures.

Production Validation

LoongCollector powers Alibaba Cloud Simple Log Service (SLS), serving billions of logs per day across 50+ regions, with deployments at the million‑instance level. Stress tests confirmed linear scaling up to 1 M+ instances, 2 k+ concurrent pipelines per node, and sub‑millisecond latency.

Chaos engineering experiments (pipeline failures, 10× traffic spikes, 90 % resource saturation, cross‑region network degradation) demonstrated resilient behavior: automatic back‑pressure, fast self‑healing via the concurrency limiter, and isolation of faulty pipelines.

Open‑Source and Community

The project is open‑source on GitHub, with the benchmark suite and source code publicly available, inviting contributions and positioning LoongCollector as a reference implementation for cloud‑native observability agents.

Overall, LoongCollector delivers ten‑fold higher maximum throughput, an 80 % reduction in resource consumption, near‑linear scalability, and robust multi‑tenant reliability, making it a compelling foundation for AI‑era observability pipelines.