How LoongCollector Redefines Log Collection: Performance, Flexibility, and Zero‑Downtime Upgrades

Background

iLogtail began in 2013 as a simple log collector for the Feitian 5K super‑computer project, evolving through Alibaba's cloud migration, SLS commercialization, and Kubernetes adoption to become a full‑featured observability data collector.

From iLogtail to LoongCollector

In 2025 iLogtail was rebranded and rebuilt as LoongCollector, introducing major upgrades in functionality, performance, and stability across all logging scenarios.

Pipeline Architecture

Each collection task is represented by a configurable pipeline. The pipeline consists of input, processor, and flusher plugins, which can be combined in various ways:

C++ Input + C++ Processor – highest performance for massive real‑time logs.

C++ Input + SPL Processor – powerful declarative data processing.

C++ Input + Golang Processor – combines C++ ingestion speed with Golang flexibility.

Golang Input + Golang Processor – supports many data sources such as Systemd, Kafka, and Windows events.

All pipelines run in a bus mode with three dedicated runner threads (Input, Processor, Flusher) that communicate via buffered queues, ensuring fair scheduling and isolation.

Hot‑Reload Isolation

LoongCollector replaces only the pipelines whose configuration changed, avoiding a global “Stop‑The‑World” pause and minimizing impact on other workloads.

Performance Improvements

Benchmarks show an average CPU reduction of 35% and memory reduction of 10% compared with iLogtail. File collection throughput increased by up to 80% in multi‑threaded scenarios, and standard‑output collection gained 100‑200% speed gains in container runtimes.

Monitoring & Self‑Health

Comprehensive instance, file‑collection, and pipeline‑detail dashboards provide real‑time visibility of CPU, memory, network, queue latency, and error metrics, with automatic alerting.

Network Isolation

Flusher threads employ adaptive rate‑limiting (AZ, Project, Logstore) based on an AIMD algorithm, isolating network failures in a single region without affecting other pipelines.

Automatic Domain Switching

SLS endpoints are probed for network quality; on degradation LoongCollector switches from intranet to public domains and back, ensuring uninterrupted data delivery.

Tag Processing Enhancements

Tag handling is unified across C++ and Go pipelines. Input plugins now manage their own tags, and a new tag‑processor plugin enables renaming, deletion, and addition of instance‑level tags consistently.

{
  "configName": "taiye-file-test-new",
  "inputs": [{
    "Type": "input_file",
    "FilePaths": ["/home/**/test.log"],
    "EnableContainerDiscovery": true,
    "CollectingContainersMeta": true,
    "ContainerFilters": {"IncludeEnv": {"aliyun_logs_taiye-file-test": "/home/test.log"}},
    "Tags": {"K8sNamespaceTagKey": "my-namespace", "ContainerIpTagKey": ""}
  }],
  "flushers": [{
    "Type": "flusher_sls",
    "Endpoint": "cn-hangzhou-intranet.log.aliyuncs.com",
    "Logstore": "taiye-file-test-new",
    "Region": "cn-hangzhou",
    "TelemetryType": "logs"
  }],
  "global": {
    "PipelineMetaTagKey": {"HOST_NAME": "taiye-123"},
    "EnableProcessorTag": true
  }
}

Zero‑Downtime Migration

Existing iLogtail configurations, checkpoints, and offsets are fully compatible; LoongCollector can replace iLogtail without data loss or service interruption.

Kubernetes Upgrade Path

DaemonSet resources are upgraded using affinity controls, allowing seamless switch from logtail‑ds to loongcollector‑ds with minimal downtime and no data duplication.

Future Directions

LoongCollector integrates Prometheus metrics, eBPF collection, and aims to become a “OneAgent” for all observability data, with more capabilities slated for release.