Master Log Processing with iLogtail SPL: From Native Plugins to Advanced Transformations

Background and SPL Introduction

With the rise of stream processing, many tools have emerged to make data handling more efficient and flexible. SLS released SPL (SLS Processing Language) to unify query, on‑node processing, and data transformation, offering both performance and flexibility. iLogtail 2.0 fully supports SPL, allowing users to replace custom plugins with concise SPL statements.

iLogtail Processing Modes

Native Plugin Mode : C++ plugins, highest performance, moderate development barrier.

Extension Plugin Mode : Go plugins, rich ecosystem, lower development barrier.

SPL Mode : C++‑based SPL engine, combines native performance with low‑code flexibility; users write SPL scripts instead of code.

Feature Comparison

Native: C++ implementation, high performance, extensive operator set.

Extension: Go implementation, good performance, flexible customization.

SPL: C++ implementation, high performance, full operator set, low‑code, no open‑source plugin required.

Advantages of iLogtail 2.0 + SPL

Unified Data Processing Language : Same language works across different scenarios, improving efficiency.

More Efficient Query Processing : SPL handles weakly structured data well; core operators are C++‑based, matching native speed.

Rich Built‑in Functions and Operators : Enables flexible composition.

Easy to Learn : Low‑code approach lets users get started quickly for log search and processing.

Practical SPL Examples

1. Regex Parsing

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+) \S+\] "(\w+) ([^\"]*)" ([\d\.]+) (\d+) (\d+) (\d+|-) "([^\"]*)" "([^\"]*)"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser
      | project-away content
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

Output:

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30",
    "method": "POST",
    "url": "/PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713184059"
}

2. Delimiter Parsing

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-csv content as ip, time, method, url, request_time, request_length, status, length, ref_url, browser
      | project-away content
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

Output:

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30 +0800",
    "method": "POST",
    "url": "PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713231487"
}

3. JSON Parsing

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | project-away content
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

Output:

{
    "url": "POST /PutData?Category=YunOsAccountOpLog HTTP/1.1",
    "ip": "10.200.98.220",
    "user-agent": "aliyun-sdk-java",
    "request": "{\"status\":\"200\",\"latency\":\"18204\"}",
    "time": "07/Jul/2022:10:30:28",
    "__time__": "1713237315"
}

4. Desensitization

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-regexp content, 'password":"(\S+)"' as password
      | extend content=replace(content, password, '******')
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

Output:

{
    "content": "{\"account\":\"1812213231432969\",\"password\":\"******\"}",
    "__time__": "1713239305"
}

5. Adding Fields

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend service='A'
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

Output:

{
    "content": "this is a test log",
    "service": "A",
    "__time__": "1713240293"
}

6. Base64 Encoding

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend content1=to_base64(cast(content as varbinary))
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

Output:

{
    "content": "this is a test log",
    "content1": "dGhpcyBpcyBhIHRlc3QgbG9n",
    "__time__": "1713318724"
}

7. MD5 Hashing

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend content1=lower(to_hex(md5(cast(content as varbinary))))
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

Output:

{
    "content": "this is a test log",
    "content1": "4f3c93e010f366eca78e00dc1ed08984",
    "__time__": "1713319673"
}

8. URL Encode/Decode and Extraction

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend encoded = url_encode(content)
      | extend decoded = url_decode(encoded)
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

Output:

{
    "content": "https://homenew.console.aliyun.com/home/dashboard/ProductAndService",
    "encoded": "https%3A%2F%2Fhomenew.console.aliyun.com%2Fhome%2Fdashboard%2FProductAndService",
    "decoded": "https://homenew.console.aliyun.com/home/dashboard/ProductAndService",
    "__time__": "1713319673"
}

9. Mathematical Functions

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend val = cast(content as double)
      | extend power_test = power(val, 2)
      | extend round_test = round(val)
      | extend sqrt_test = sqrt(val)
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

Output:

{
    "content": "4",
    "val": 4.0,
    "power_test": 16.0,
    "round_test": 4.0,
    "sqrt_test": 2.0,
    "__time__": "1713319673"
}

10. Comparison and Logical Operators

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | extend compare_result = cast(num1 as double) > cast(num2 as double) AND cast(num2 as double) > cast(num3 as double)
flushers:
  - Type: flusher_stdout
    OnlyStdout: true

Output:

{
    "compare_result": "true",
    "content": "{\"num1\":199,\"num2\":10,\"num3\":9}",
    "num1": "199",
    "num2": "10",
    "num3": "9",
    "__time__": "1713319673"
}

For a complete list of SPL functions and capabilities, refer to the official SLS user guide at https://help.aliyun.com/zh/sls/user-guide/function-overview. Users are encouraged to contribute additional iLogtail SPL use cases.