Why iLogtail Needed a Complete Architecture Overhaul and How It Was Done

Overview

iLogtail is a high‑performance lightweight observability data collector from Alibaba Cloud, used in servers, containers and embedded environments, processing tens of petabytes of logs, metrics and traces daily.

Architecture Evolution

Single‑File Collection Stage

Only collects log files.

Assumes a single format per log.

Outputs only to the SLS service.

Implemented entirely in C++ with a monolithic design that limited extensibility.

Golang Plugin Extension Stage

Introduced a Golang plugin system to support diverse inputs, outputs and processing capabilities.

Each pipeline runs independently.

Plugins can be inputs, processors or flushers.

However, combination between C++ and Golang components remained limited.

Why Refactor?

The original architecture restricted input/output combinations, made C++ code hard to maintain, and prevented third‑party storage use while the open‑source version exposed many hidden issues.

Refactor Goals

Replace the internal data model with a generic one supporting Log, Metric and Trace.

Plugin‑ize all C++ functionality.

Introduce a pipeline concept in C++.

Unify configuration format for commercial and open‑source versions.

Improve hot‑loading of configurations.

Simplify commercial code integration.

Practices

Data Model Generalization

Define PipelineEventGroup and PipelineEvent hierarchy to replace the LogGroup protobuf.

message Log {
    required uint32 Time = 1;
    message Content {
        required string Key = 1;
        required string Value = 2;
    }
    repeated Content Contents = 2;
    repeated string values = 3;
    optional fixed32 Time_ns = 4;
}

message LogTag {
    required string Key = 1;
    required string Value = 2;
}

message LogGroup {
    repeated Log Logs = 1;
    optional string Category = 2;
    optional string Topic = 3;
    optional string Source = 4;
    optional string MachineUUID = 5;
    repeated LogTag LogTags = 6;
}

The new model supports Log, Metric and Span events and can be serialized for any downstream system.

Plugin Abstraction

Define abstract base Plugin and derived interfaces Input, Processor, Flusher. Implement native C++ plugins such as ProcessorSplitLogStringNative, ProcessorParseJsonNative, ProcessorFilterRegexNative, etc., and treat Golang plugins uniformly.

Pipeline Definition

Class Pipeline holds vectors of input, processor and flusher instances, a processing queue, and a context. Lifecycle methods Init, Start, Process, Stop manage the flow from inputs through processors to flushers.

Configuration Management

Adopt a unified file‑per‑config layout, support JSON for commercial and YAML for open‑source, and monitor changes via ConfigWatcher. Hot‑load uses PipelineManager::UpdatePipelines to add, modify or remove pipelines without disrupting running ones.

Remote Config and Enterprise Integration

Introduce ConfigProvider hierarchy ( EnterpriseConfigProvider, CommonConfigProvider) and use compile‑time __ENTERPRISE__ switches to embed commercial features without file replacement, allowing clean separation of open‑source and enterprise code.

Reflection

The refactor took over half a year, required careful dependency analysis, extensive testing, and adherence to design‑pattern literature to ensure code quality and maintainability.

Conclusion

The new architecture gives iLogtail a modular, extensible pipeline, a unified data model, and smoother commercial/open‑source coexistence, positioning it as a modern observability collector.