iLogtail SPL vs Logstash: Faster, Lighter, More Flexible Log Processing

Introduction

In the data‑driven era, log collection and processing tools are essential for system stability and operational efficiency. As data volumes grow and architectures become more complex, traditional log tools face challenges in performance, flexibility, and usability. Logstash has been widely used, but a new generation solution—iLogtail—has emerged.

iLogtail 2.0 recently added a new SPL processing mode, enhancing log handling capabilities. This article explores why to choose iLogtail and the unique advantages of its SPL mode compared with Logstash.

iLogtail & SPL Overview

iLogtail is an open‑source log collector from Alibaba, optimized for large‑scale data and cloud‑native scenarios. It offers low latency, high efficiency, lightweight deployment, and automated configuration, and can run on physical machines, VMs, or Kubernetes clusters to collect logs, traces, and metrics.

Version 2.0 adds SPL (Stream Processing Language), a flow‑oriented language inspired by Unix pipelines, designed for unstructured log data. Logs are collected, processed through a series of SPL operators, and then forwarded downstream.

SPL leverages high‑performance techniques such as SIMD parallelism, delivering flexibility without sacrificing speed.

iLogtail SPL vs Logstash Filter

Logstash’s filter plugins (e.g., grok, date, mutate) provide powerful log transformation capabilities. However, compared with iLogtail’s SPL, the two differ in functionality support and scenario performance.

Functionality Support

According to official documentation, Logstash offers 49 filter plugins covering most common log processing needs.

SPL provides 8 core data‑processing instructions (including regex and JSON parsing) and 110 SQL‑compatible commands, with ongoing development.

Configuration Examples

Regex + Time Parsing (nginx access log)

142.207.88.67 - - [18/Jun/2024:12:14:26 +0800] "DELETE http://www.districtdot-com.biz/syndicate HTTP/1.1" 289 3715 "http://www.chiefscalable.biz/webservices" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_2_1 like Mac OS X; en-US) AppleWebKit/534.46.7 (KHTML, like Gecko) Version/5.0.5 Mobile/8B119 Safari/6534.46.7"

Logstash relies on the Grok plugin for regex, while SPL handles it natively; the user experience is similar for this basic case.

Conditional Field Add/Delete/Filter (simple JSON log)

{"url": "POST /PutData HTTP/1.1", "user-agent": "aliyun-sdk-java"}

Logstash’s configuration language is more verbose and less suited for complex data manipulation compared with SPL’s SQL‑like syntax.

Nested JSON Extraction

2024-06-24 12:26:04.063 INFO 24 --- [traceId=edda5daxxxxxxxxxcfa3387d48][ xnio-1 task-1] c.g.c.gateway.filter.AutoTestFilter : {"traceId":"edda5da8xxxxxxxxxxxxxxxxxxx387d48","headers":[{"x-forwarded-proto":"http,http","x-tenant-id":"123","x-ca-key":"a62d5xxxxxxxxxxxxxxxxxxxxxxxxb1cff8637","x-forwarded-port":"80,80","x-forwarded-for":"10.244.2.0","x-ca-client-ip":"10.244.2.0","x-product-code":"xxxxx","authorization":"bearer 0ed29xxxxxxxxxxxxxxxxxxxxxxxxx71899","x-forwarded-host":"gatxxxxxxxxx.gm","x-forwarded-prefix":"/xxxxxx","trace-id":"edda5da8278xxxxxxxxxxxxxxxxxxx49cfa3387d48","x-ca-api-id":"1418470181321347075","x-ca-env-code":"TEST"}],"appName":"超级管理员","responseTime":15,"serverName":"test-server","appkey":"a62d54b6bxxxxxxxxxxxxxxxxxxx37","time":"2021-08-01 12:26:04.062","responseStatus":200,"url":"/test/v4/orgs/123/list-children","token":"bearer 0ed29c72-0d68-4e13-a3f3-c77e2d971899"}

SPL supports JsonPath extraction, making nested JSON handling concise, whereas Logstash requires multiple plugins and complex pipelines.

Scenario Performance

Resource usage is critical for a log collector. Tests compared iLogtail and Logstash on a Linux host (8 CPU, 16 GB RAM) using default configurations.

Test environment: 8‑core, 16 GB Ubuntu 22.04

Versions: iLogtail 2.0.4, Logstash 8.14.1

Scenarios: (1) Regex + time parsing, (2) Conditional field ops, (3) Multi‑level JSON extraction

Data rates: 1 MB/s, 5 MB/s, 10 MB/s for 3 minutes

All logs were discarded after processing to isolate collector overhead.

Scenario 1 – Regex + Time Parsing

Logstash shows higher startup latency, CPU, and memory usage. Its CPU consumption at 1 MB/s matches iLogtail’s at 10 MB/s, and its CPU usage fluctuates more.

Scenario 2 – Conditional Field Ops

After log generation stops, Logstash continues processing, indicating a bottleneck and noticeable delay. iLogtail processes the same workload with near‑zero delay and low resource usage, handling up to 50 MB/s versus Logstash’s 8 MB/s limit.

Scenario 3 – Nested JSON Extraction

Logstash’s CPU shows similar spikes as in Scenario 1, but overall performance remains inferior to iLogtail.

Analysis

Resource Consumption – iLogtail’s average memory stays around 40 MB (max ~50 MB), while Logstash starts at ~900 MB and can exceed 1.2 GB. CPU usage for iLogtail is stable; Logstash exhibits large spikes and irregular patterns.

Real‑time Capability – iLogtail reacts to log generation immediately, whereas Logstash introduces a ~10 second delay before CPU usage rises.

Performance – In high‑throughput tests, iLogtail processes up to 50 MB/s with low latency, six times faster than Logstash, thanks to its C++ core and SIMD optimizations. Logstash’s Ruby‑based plugins limit parallelism.

Extensibility – Logstash allows custom Ruby or Java plugins; iLogtail currently lacks user‑defined extensions but plans to support UDFs and offers C++/Go plugin interfaces.

Conclusion

iLogtail’s SPL mode demonstrates clear advantages over Logstash’s filter plugins in functionality, resource efficiency, and throughput. While Logstash provides a broad plugin ecosystem, its configuration can be cumbersome for complex logic and nested JSON handling. iLogtail’s integrated SPL, SQL compatibility, and lightweight footprint make it a compelling choice for enterprises seeking high‑performance, resource‑friendly log processing in cloud‑native environments.

Relevant links:

iLogtail GitHub

Logstash Filter Plugins Documentation

iLogtail SPL Overview