LoongCollector: 10× Throughput, 80% Resource Savings in Cloud‑Native Observability

Introduction

Over more than a decade of large‑scale cloud workloads, Alibaba has evolved LoongCollector (formerly iLogtail) into a cloud‑native unified collector that can handle daily data volumes at the hundred‑petabyte level while reducing resource consumption by about 80%.

Performance Benchmarks

In a reproducible benchmark on an Alibaba Cloud ECS g7 instance (32 vCPU, 64 GB RAM, ESSD PL3 1500 GiB), LoongCollector achieved a maximum throughput of 546 MB/s for single‑line logs, far surpassing FluentBit (36 MB/s), Vector (38 MB/s) and Filebeat (9 MB/s). CPU usage for the same workload was only 3.40 % compared to 12.29 % for FluentBit and 35.80 % for Vector, representing up to a 90% reduction in CPU load.

Benchmark tables (excerpt):

Metric          | LoongCollector | FluentBit | Vector | Filebeat
---|---|---|---|---
Single‑line 512 B | 546 MB/s (3.40 % CPU) | 36 MB/s (12.29 % CPU) | 38 MB/s (35.80 % CPU) | 9 MB/s (performance insufficient)
Multi‑line 512 B | 238 MB/s (5.82 % CPU) | 24 MB/s (28.35 % CPU) | 22 MB/s (55.99 % CPU) | 6 MB/s (performance insufficient)
Regex 512 B     | 68 MB/s (14.20 % CPU) | 19 MB/s (37.32 % CPU) | 12 MB/s (43.90 % CPU) | not supported

Key Architectural Techniques

Zero‑Copy String Handling : Logs are stored once in a shared SourceBuffer and parsed using string_view, eliminating the typical four string copies of traditional agents.

Lock‑Free Event Pools : Each processing thread uses a thread‑local lock‑free pool for PipelineEventGroup objects, with a double‑buffer pool for cross‑thread handoff, reducing allocation overhead by about 90%.

Zero‑Copy Serialization : Events are written directly in Protobuf wire format without constructing intermediate objects, cutting serialization CPU from 12.5 % to 5.8 % (≈54% reduction) and memory copies from three to one (≈67% reduction).

Reliability and Isolation

LoongCollector employs per‑pipeline bounded queues with high/low watermark feedback to provide back‑pressure and prevent a single slow downstream from blocking the entire system. Multi‑tenant isolation ensures each pipeline has dedicated memory and CPU resources, avoiding resource contention.

Scheduling uses priority‑aware round‑robin: high‑priority pipelines are always served first, while lower‑priority pipelines share remaining capacity fairly, preventing starvation.

Adaptive Concurrency (Self‑Healing Network)

Each destination endpoint has an AIMD‑inspired concurrency limiter. When failure rates rise, concurrency is reduced multiplicatively (e.g., ×0.5), and when success rates are high, it increases additively, providing rapid “stop‑bleeding” and graceful recovery without overwhelming the network.

Production Validation

LoongCollector powers Alibaba Cloud Simple Log Service (SLS), handling over a hundred petabytes of daily logs across more than 50 regions, deployed on millions of instances for Fortune 500 customers in finance, retail, and manufacturing. Large‑scale chaos tests (pipeline failures, 10× traffic spikes, 90% resource saturation) confirm its scalability, network resilience, and fault isolation.

Conclusion

The combination of zero‑copy processing, lock‑free pooling, multi‑tenant isolation, and adaptive concurrency gives LoongCollector a ten‑fold throughput advantage and substantial resource savings while maintaining enterprise‑grade reliability, making it a solid foundation for observability in the AI and FinOps era.