How We Boosted Multi-line Log Collection Speed from 90 MB/s to 350 MB/s

Problem Background

In the digital era, log data has become a key resource for IT operations and business analysis. As business scale expands, log volume grows explosively, creating huge challenges for log collection and processing systems. A recent customer case illustrated this: the client generated logs at 200 MB/s, but iLogtail could only collect 90 MB/s, causing nearly an hour of delay.

Massive multi‑line logs required regex parsing, demanding high system resources.

High collection latency directly impacted business‑critical analytics.

Even after tuning iLogtail thread counts, the collection speed remained far below log generation speed.

The client planned to increase write traffic to 300 MB/s, further stressing the system.

Business workload already consumed most CPU resources, leaving little headroom for iLogtail.

Technical Challenges

Performance bottleneck : iLogtail used boost::regex_match for full‑line matching, which scales linearly with log length.

Resource consumption vs. business conflict : Achieving target throughput required 16 threads, heavily loading the CPU.

Diminishing returns on thread scaling : Adding more threads yielded marginal gains after a certain point.

Optimization Process and Results

After analyzing test logs, we identified the core issue as the full‑line regex match. We replaced boost::regex_match with boost::regex_search combined with the boost::match_continuous flag to perform prefix‑only matching.

bool BoostRegexSearch(const char* buffer, size_t size, const boost::regex& reg, string& exception) {
    // ...
    if (boost::regex_search(buffer, buffer + size, what, reg, boost::match_continuous)) {
        return true;
    }
    // ...
}

This change eliminated the linear cost of matching the entire line. Benchmarking showed that while boost::regex_match time grew with log length, boost::regex_search remained stable.

Performance after optimization:

Collection speed increased to >300 MB/s, fully meeting the client’s requirement.

Thread count reduced from 16 to 8, significantly lowering CPU usage.

The new IngestProcessor solution allowed a single thread to achieve 320 MB/s with minimal resources.

IngestProcessor – Solving Resource Constraints

To further reduce client‑side resource pressure, Alibaba Cloud Log Service introduced the IngestProcessor. It processes logs in the cloud, moving heavy computation away from the client.

CPU usage on the client drops from 16 cores to 1 core (≈94% reduction).

Maintains a high collection rate of ~320 MB/s.

Provides rich data‑processing capabilities such as field extraction, enrichment, masking, and filtering.

Conclusion

The optimized iLogtail, combined with IngestProcessor, can reliably handle 300 MB/s of multi‑line logs with regex extraction using only 8 threads, or even a single thread in the cloud‑side solution. This demonstrates that targeted algorithmic improvements (prefix‑only regex) and off‑loading processing to the cloud can dramatically boost log‑collection performance while drastically reducing client resource consumption.