How SPL’s New Operators Supercharge Log‑to‑Metric Processing in Cloud‑Native Environments

Background

Since its launch, the Search Processing Language (SPL) of Alibaba Cloud Log Service has become a preferred tool for developers and enterprises to perform high‑performance data analysis. Continuous business expansion and growing technical demands have driven SPL to evolve, delivering stronger and more flexible data processing capabilities.

New Operator Features

The latest SPL update adds three powerful operators— pack-fields, log-to-metric, and metric-to-metric —that streamline the conversion from raw logs to structured data and then to time‑series metrics, expanding use cases such as observability analysis and time‑series forecasting.

pack-fields

pack-fields

(e_pack_fields) aggregates selected fields into a JSON object, achieving extreme data density compression.

It introduces intelligent field trimming via the -ltrim='xxx' parameter, allowing dynamic removal of field prefixes (e.g., trimming mdc_ from mdc_key1 to key1).

Compatibility is maintained with existing operators like parse-kv, forming a complete data‑regularization pipeline.

log-to-metric

Transforms unstructured logs into time‑series metric logs, automatically inferring numeric fields to preserve value precision.

Supports one‑click formatting with the key#$#value pattern, standardizing label encoding.

Wildcard matching ( -wildcard) enables pattern‑based field capture (e.g., request* matches all fields starting with request).

metric-to-metric

Provides secondary processing of metric data, supporting label addition, deletion, and renaming via -add_labels, -del_labels, and -rename_label.

Includes a format cleaner that removes illegal characters and normalizes key‑value pairs.

Offers dimension distillation to filter and retain core metrics, reducing storage and query overhead.

Code Examples

# 场景示例：日志字段聚合
* | parse-kv -prefix="mdc_" -regexp content,'(\w+)=(\w+)'
| pack-fields -include='mdc_.*' -ltrim='mdc_' as mdc

# 输入数据
__time__: 1614739608
rt: 123
qps: 10
host: myhost
# SPL语句
* | log-to-metric -names='["rt","qps"]' -labels='["host"]'
# 输出两条Metric日志
__labels__:host#$#myhost
__name__:rt
__time_nano__:1614739608
__value__:123
__labels__:host#$#myhost
__name__:qps
__time_nano__:1614739608
__value__:10

# 输入数据
__labels__:host#$#myhost|qps#$#10|asda_cc#$#j|schema#$#
__name__:rt
__time_nano__:1614739608
__value__:123
# SPL语句
* | metric-to-metric -format
# 输出数据
__labels__:asda_cc#$#j|host#$#myhost|qps#$#10
__name__:rt
__time_nano__:1614739608
__value__:123

Performance Evaluation

Performance was measured by comparing the new SPL operators against the legacy DSL implementation. Tests included:

Data simulation with mock datasets of comparable memory size to ensure input consistency.

End‑to‑end benchmarks focusing on critical modules ( log-to-metric and pack-fields), covering the full pipeline from ingestion to output.

Results show:

Overall end‑to‑end throughput increased by up to 7.17× for log-to-metric and 37.23× for pack-fields.

Processing speed improvements of 27.8× (log‑to‑metric) and 51.52× (pack‑fields) were observed, eliminating the bottlenecks of the previous implementation.

Conclusion

The SPL update delivers three core breakthroughs: extreme performance and stability through a C++‑backed processing engine, a dramatically improved user experience with SQL‑like syntax, one‑click formatting, and wildcard matching, and enhanced observability capabilities that bridge logs to metrics for comprehensive monitoring. Future work will continue to focus on generality, performance, and product features to provide ever‑stronger technical support for users.