What’s New in iLogtail 2.0? A Deep Dive into the Updated Pipeline Architecture

Background

iLogtail 1.x was designed for simple file‑log ingestion to Log Service, using a flat configuration and a monolithic process. This architecture limited extensibility, prevented multiple processing methods per log format, and tightly coupled modules to Log Service concepts such as Logstore.

Pipeline‑Oriented Configuration (iLogtail 2.0)

The new design introduces a pipeline model composed of inputs , processors , and flushers . Each module can contain one or more plugins that perform specific functions.

Input plugins acquire data from sources (e.g., files, containers).

Processor plugins parse or transform logs. They are divided into native (high‑performance C++) and extended (more feature‑rich) types.

Flusher plugins deliver processed data to target stores such as Log Service.

Example JSON pipeline configuration:

{
  "configName": "test-config",
  "inputs": [{"Type": "file_log", "FilePaths": "/var/log/test.log"}],
  "processors": [{"Type": "processor_parse_json_native", "SourceKey": "content"}],
  "flushers": [{"Type": "flusher_sls", "Logstore": "test_logstore"}]
}

The legacy flat configuration remains supported for backward compatibility.

OpenAPI Endpoints for Pipeline Management

CreateLogtailPipelineConfig

UpdateLogtailPipelineConfig

GetLogtailPipelineConfig

DeleteLogtailPipelineConfig

ListLogtailPipelineConfig

Detailed specifications are available in the OpenAPI documentation.

Kubernetes CRD Upgrade

The Custom Resource Definition now uses the pipeline model, is cluster‑scoped, and the CRD name serves as the configuration identifier, avoiding naming conflicts.

apiVersion: log.alibabacloud.com/v1alpha1
kind: ClusterAliyunLogConfig
metadata:
  name: test-config
spec:
  project:
    name: test-project
  logstore:
    name: test-logstore
  machineGroup:
    name: test-machine_group
  config:
    inputs:
      - Type: input_file
        FilePaths:
          - /var/log/test.log
    processors:
      - Type: processor_parse_json_native
        SourceKey: content

Flexible Processor Composition

Native processors can be combined arbitrarily, and extended processors may follow native ones. Extended processors must appear after all native processors.

Example: parse JSON, then apply a regex, then drop a field. <code>{ "configName": "test-config", "inputs": [...], "processors": [ {"Type": "processor_parse_json_native", "SourceKey": "content"}, {"Type": "processor_parse_regex_native", "SourceKey": "detail", "Regex": "(\\S)+\\s(\\S)+\\s(.*)", "Keys": ["ip","method","status"]}, {"Type": "processor_drop", "DropKeys": ["module"]} ], "flushers": [...] } </code>

SLS Processing Language (SPL)

SPL provides a low‑code, pipe‑driven syntax for both structured and unstructured data. It supports SQL‑style operators, built‑in parsers, and functions.

*
| extend latency=cast(latency as BIGINT)
| where status='200' AND latency>100

*
| project-csv -delim='^_^' content as time, body
| project-regexp body, '(\S+)\s+(\w+)' as msg, user

Fine‑Grained Parsing Controls

Native parsers expose three additional parameters:

KeepingSourceWhenParseFail – retain the original field when parsing fails.

KeepingSourceWhenParseSucceed – retain the original field when parsing succeeds.

RenameSourceKey – rename the retained source field.

To keep failed parses under the field raw : KeepingSourceWhenParseFail: true RenameSourceKey: raw

Nanosecond‑Level Timestamp Support

iLogtail 2.0 can attach nanosecond precision directly to the built‑in __time__ field. Add .%f to the source time format and set EnableTimestampNanosecond: true in global parameters.

Example: parsing 2024-01-23T14:00:00.745074 yields nanosecond‑precision timestamps.

Observability Enhancements

Each pipeline configuration emits complete metrics, enabling per‑project/logstore analysis.

Every plugin provides its own metrics, allowing full topology monitoring.

C++ native plugins expose detailed counters for performance tuning.

Performance Benchmarks

Single‑line log: CPU 1.06 cores, Memory 33 MB, Throughput 400 MB/s.

Multi‑line log: CPU 1.04 cores, Memory 33 MB, Throughput 150 MB/s.

Compatibility Notes

During upgrade, the service automatically converts legacy configurations to the new pipeline format. Not all legacy features map back; parameters such as PreciseTimestampKey and AdjustingMicroTimezone are deprecated.

Tag handling differs between commercial and open‑source editions. The commercial edition places tags in a dedicated field by default; the open‑source edition can retain the old behavior via the global parameter UsingOldContentTag: true.

Summary

iLogtail 2.0 modernizes log collection with a flexible pipeline model, SPL low‑code processing, finer parsing controls, nanosecond‑precision timestamps, and comprehensive observability, providing a cloud‑native solution for complex observability workloads.