Cloud Native 20 min read

Master Log Processing with iLogtail SPL: From Native Plugins to Advanced Transformations

This article introduces SLS's SPL (SLS Processing Language) for iLogtail, compares its three processing modes, details feature differences, highlights the advantages of iLogtail 2.0 + SPL, and provides step‑by‑step SPL examples covering regex, delimiter, JSON parsing, desensitization, field addition, encoding, mathematical operations, URL handling, and logical comparisons.

Alibaba Cloud Observability

Nov 13, 2024

Master Log Processing with iLogtail SPL: From Native Plugins to Advanced Transformations

With the rise of stream processing, many tools and languages have emerged to make data handling more efficient, flexible, and user‑friendly. In this context, SLS introduced SPL (SLS Processing Language) to unify query, edge processing, and data transformation syntax, ensuring flexibility. iLogtail, a log and time‑series data collector, fully supports SPL in version 2.0. This article reviews processing plugins, explains how to write SPL statements, and guides migration from native plugin processing to SPL mode.

Processing Modes

Native Plugin Mode: Native plugins implemented in C++, offering the highest performance.

Extension Plugin Mode: Plugins implemented in Go, providing a rich ecosystem and sufficient flexibility.

SPL Mode: iLogtail 2.0 adds SPL processing capabilities, combining performance and flexibility; users only need to write SPL statements to leverage its computation power.

Feature Comparison

Native Plugin

C++ implementation

High performance, minimal resource overhead

Comprehensive operator capabilities

Medium development barrier (C++)

Highly customizable

Extension Plugin

Go implementation

High performance, low resource overhead

Comprehensive operator capabilities

Low development barrier (Go)

Highly customizable

SPL

C++ implementation (core operators)

High performance, low resource overhead

Full operator set

Flexible composition

Not yet open‑source; solves most problems without coding

Advantages of iLogtail 2.0 + SPL

Unified Data Processing Language: The same language can be used across different scenarios, improving processing efficiency.

More Efficient Query Processing: SPL is friendly to weakly structured data and its core operators are implemented in C++, approaching native performance.

Rich Tools and Functions: SPL provides many built‑in functions and operators for flexible composition.

Simple and Easy to Learn: As a low‑code language, SPL allows rapid onboarding for log search and processing.

Example Scenarios

Regex Parsing

Input (Nginx format):

127.0.0.1 - - [07/Jul/2022:10:43:30 +0800] "POST /PutData?Category=YunOsAccountOpLog" 0.024 18204 200 37 "-" "aliyun-sdk-java"

Original plugin configuration:

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_parse_regex_native
    SourceKey: content
    Regex: ([\d\.]+) \S+ \S+ \[(\S+) \S+\] "(\w+) ([^\"]*)" ([\d\.]+) (\d+) (\d+) (\d+|-) "([^\"]*)" "([^\"]*)"
    Keys:
      - ip
      - time
      - method
      - url
      - request_time
      - request_length
      - status
      - length
      - ref_url
      - browser
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

SPL configuration:

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+) \S+\] "(\w+) ([^\"]*)" ([\d\.]+) (\d+) (\d+) (\d+|-) "([^\"]*)" "([^\"]*)"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser
      | project-away content
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

Output:

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30 +0800",
    "method": "POST",
    "url": "/PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713319673"
}

Delimiter Parsing

Input (comma‑separated):

127.0.0.1,07/Jul/2022:10:43:30 +0800,POST,PutData Category=YunOsAccountOpLog,0.024,18204,200,37,-,aliyun-sdk-java

Original plugin:

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_parse_delimiter_native
    SourceKey: content
    Separator: ","
    Quote: '"'
    Keys:
      - ip
      - time
      - method
      - url
      - request_time
      - request_length
      - status
      - length
      - ref_url
      - browser
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

SPL:

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-csv content as ip, time, method, url, request_time, request_length, status, length, ref_url, browser
      | project-away content
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

Output:

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30 +0800",
    "method": "POST",
    "url": "PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713319673"
}

JSON Parsing

Input JSON:

{"url":"POST /PutData?Category=YunOsAccountOpLog HTTP/1.1","ip":"10.200.98.220","user-agent":"aliyun-sdk-java","request":"{\"status\":\"200\",\"latency\":\"18204\"}","time":"07/Jul/2022:10:30:28","__time__":"1713237315"}

Original plugin:

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_parse_json_native
    SourceKey: content
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

SPL:

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | project-away content
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

Output:

{
    "url": "POST /PutData?Category=YunOsAccountOpLog HTTP/1.1",
    "ip": "10.200.98.220",
    "user-agent": "aliyun-sdk-java",
    "request": "{\"status\":\"200\",\"latency\":\"18204\"}",
    "time": "07/Jul/2022:10:30:28",
    "__time__": "1713237315"
}

Regex Parsing + Timestamp Extraction

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+)\] "(\w+) ([^\"]*)" ([\d\.]+) (\d+) (\d+) (\d+|-) "([^\"]*)" "([^\"]*)"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser
      | extend ts = date_parse(time, '%Y-%m-%d %H:%i:%S')
      | extend __time__ = cast(to_unixtime(ts) as INTEGER)
      | project-away ts, content
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

Regex Parsing + Filtering

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+) \S+\] "(\w+) ([^\"]*)" ([\d\.]+) (\d+) (\d+) (\d+|-) "([^\"]*)" "([^\"]*)"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser
      | project-away content
      | where regexp_like(method, '^(POST|PUT)$') and regexp_like(status, '^200$')
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

Desensitization

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-regexp content, 'password":"(\S+)"' as password
      | extend content = replace(content, password, '******')
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

Adding Fields

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend service = 'A'
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

JSON Parsing + Drop Field

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | project-away content
      | project-away key1
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

JSON Parsing + Rename Field

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | project-away content
      | project-rename new_key1 = key1
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

JSON Filtering

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | project-away content
      | where regexp_like(ip, '10\..*') and regexp_like(method, 'POST') and not regexp_like(browser, 'aliyun.*')
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

Dictionary Mapping

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | project-away content
      | extend _processed_ip_ = CASE
        WHEN _ip_ = '127.0.0.1' THEN 'LocalHost-LocalHost'
        WHEN _ip_ = '192.168.0.1' THEN 'default login'
        ELSE 'Not Detected'
      END
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

String Replacement

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend content = replace(content, 'how old are you?', '')
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

Base64 Encoding

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend content1 = to_base64(cast(content as varbinary))
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

MD5 Hashing

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend content1 = lower(to_hex(md5(cast(content as varbinary))))
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

Mathematical Computation

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend val = cast(content as double)
      | extend power_test = power(val, 2)
      | extend round_test = round(val)
      | extend sqrt_test = sqrt(val)
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

URL Encode/Decode

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend encoded = url_encode(content)
      | extend decoded = url_decode(encoded)
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

URL Extraction

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | extend host = url_extract_host(content)
      | extend query = url_extract_query(content)
      | extend path = url_extract_path(content)
      | extend protocol = url_extract_protocol(content)
      | extend port = url_extract_port(content)
      | extend param = url_extract_parameter(content, 'accounttraceid')
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

Comparison & Logical Operators

enable: true
inputs:
  - Type: input_file
    FilePaths:
      - /workspaces/ilogtail/debug/simple.log
processors:
  - Type: processor_spl
    Script: |
      *
      | parse-json content
      | extend compare_result = cast(num1 as double) > cast(num2 as double) AND cast(num2 as double) > cast(num3 as double)
    flushers:
      - Type: flusher_stdout
        OnlyStdout: true

For more capabilities, refer to the official documentation: https://help.aliyun.com/zh/sls/user-guide/function-overview . Feel free to contribute additional iLogtail SPL use cases!

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.

iLogtail data transformation Log Processing SPL

Written by

Alibaba Cloud Observability

Driving continuous progress in observability technology!

0 followers

Reader feedback

How this landed with the community

Rate this article

Was this worth your time?

Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.

Processing Modes

Feature Comparison

Advantages of iLogtail 2.0 + SPL

Example Scenarios

Regex Parsing

Delimiter Parsing

JSON Parsing

Regex Parsing + Timestamp Extraction

Regex Parsing + Filtering

Desensitization

Adding Fields

JSON Parsing + Drop Field

JSON Parsing + Rename Field

JSON Filtering

Dictionary Mapping

String Replacement

Base64 Encoding

MD5 Hashing

Mathematical Computation

URL Encode/Decode

URL Extraction

Comparison & Logical Operators

Alibaba Cloud Observability

How this landed with the community

Was this worth your time?

0 Comments

Advantages of iLogtail 2.0 + SPL