Optimizing Global Log Collection with Alibaba Cloud iLogtail and LoongCollector

Background and Challenges

As enterprises expand globally, collecting logs from overseas data centers, other cloud providers, and Alibaba Cloud regions into a unified SLS project becomes critical for observability, troubleshooting, and compliance. The main challenges are network quality (high latency, jitter, packet loss), cost control (public‑internet egress fees), high availability (avoiding single‑point failures), and compliance with local data‑security regulations.

Core Log Collection Agents

Alibaba Cloud provides two official agents: iLogtail and LoongCollector . Both are lightweight C++ agents with low CPU/memory impact and support a wide range of data sources (files, containers, syslog, HTTP). They include a powerful Processor plugin framework for parsing, filtering, and masking logs before transmission.

Key Advantages

Lightweight : Minimal resource consumption.

Universal Collection : Supports file logs, container stdout/stderr, syslog, HTTP, etc.

Processor Plugins : Parse (JSON, regex), filter, and mask logs on the agent side, reducing unnecessary data transfer.

Compression : Built‑in LZ4 compression ( lz4) significantly reduces network bandwidth.

Reliable Transmission : Local disk cache, retry, flow‑shaping, and breakpoint resume ensure "no loss, no duplication".

Multi‑Target Output : One agent can send the same data to multiple SLS endpoints for disaster recovery or migration.

Cloud‑Native Integration : Seamless deployment on ECS, ACK/ASK, and Kubernetes (DaemonSet, Sidecar, CRD).

Why Prefer LoongCollector

Higher reliability with network‑exception isolation for each target endpoint.

When one region (e.g., Singapore) experiences network failures, LoongCollector isolates that path while continuing to send logs to healthy regions (e.g., Hangzhou), preventing a single failure from blocking all traffic.

Recommended version: 3.0.9+ (iLogtail: 2.1.7+ ).

Network Access Options

Direct Public Internet : Simplest configuration, but incurs egress fees and suffers from unstable cross‑border latency.

Global Acceleration (GA) : Uses Alibaba Cloud PoP to improve latency and packet loss; higher cost due to acceleration traffic.

Same‑Region VPC Private Network : Best performance, lowest cost, and highest security (no public traffic).

Hybrid/Multi‑Cloud via Dedicated Line, CEN, or VPN : Private network with the highest reliability; highest cost and complexity.

Cost‑Optimization Strategies

Use CloudLens for SLS to diagnose abnormal log volume or public‑network traffic and pinpoint inefficient log sources.

Data Compression : Both agents enable LZ4 compression by default, typically achieving 5‑10× reduction in bandwidth.

Log Filtering : Configure Processor plugins or SPL scripts to drop low‑value logs (debug, health‑check) before transmission.

Data Compression Details

The agents use the high‑speed, lossless LZ4 algorithm, which provides fast compression/decompression with low CPU overhead, making it ideal for real‑time log streams. Compression is enabled by default; no extra configuration is required.

Log Filtering Techniques

Use native filter plugins or custom extensions to define match rules, or write SPL queries for complex parsing and filtering. This reduces storage and transmission costs by eliminating unnecessary log entries.

Multi‑Region Log Distribution

Agents can be configured with multiple endpoints, allowing the same log data to be sent to different SLS projects in different regions. This supports scenarios such as sending system logs to a central Shanghai project while sending application logs to a Singapore project.

Configuration Examples

{
  "config_server_address": "http://logtail.ap-southeast-1-intranet.log.aliyuncs.com",
  "config_server_address_list": ["http://cn-shanghai.log.aliyuncs.com"],
  "data_server_list": [
    {"cluster": "ap-southeast-1", "endpoint": "ap-southeast-1-intranet.log.aliyuncs.com"},
    {"cluster": "cn-shanghai", "endpoint": "cn-shanghai.log.aliyuncs.com"}
  ]
}

For LoongCollector the equivalent JSON uses primary_region, config_servers, and data_servers arrays with region and endpoint definitions.

Verification and Monitoring

After deployment, verify that both Project A and Project B machine‑group heartbeats are OK and that logs appear in the respective Logstores. Use CloudLens for SLS to monitor write traffic, compression ratios, and detect anomalies. Additional SLS dashboards (overall status, file‑collection health, exception alerts) provide comprehensive observability.

Best‑Practice Summary

Assess deployment environment, log volume, latency, security, and budget to choose the optimal network path (prefer same‑region VPC).

Balance cost and performance; use public Internet only when simplicity outweighs expense.

Enable full monitoring with CloudLens and SLS alerts for write success rate, latency, and agent health.

Adopt LoongCollector for higher reliability, especially for multi‑region double‑write scenarios.

Leverage double‑write for seamless migration and disaster recovery without service interruption.

Consult Alibaba Cloud support tickets for any unresolved issues.