How New SPL Operators Supercharge Log Processing Performance

Background

Since its launch, the SPL (Search Processing Language) of Log Service has been a preferred tool for efficient data analysis, and continuous iteration adds stronger, more flexible data processing capabilities.

New Operator Features

pack-fields

: Evolves the e_pack_fields operator to aggregate fields into compact JSON objects, reducing data density and supporting intelligent field trimming with regex extraction. log-to-metric: Transforms unstructured logs into time‑series metrics using a hash‑based write path for balanced sharding and improved query performance. metric-to-metric: Provides second‑stage processing for time‑series data, enabling precise label addition, deletion, renaming, and format purification to eliminate tag pollution and naming conflicts.

Key Improvements

Intelligent Field Trimming : Dynamically filter field prefixes (e.g., -ltrim='mdc_') to extract KV structures.

Compatibility Evolution : Seamlessly integrates with operators like parse-kv for a complete data regularization pipeline.

Wildcard Matching : Pattern‑based field capture using -wildcard (e.g., request*).

Performance Enhancements

The new SPL operators are implemented in high‑efficiency C++ with algorithmic optimizations, achieving up to 10× overall throughput improvement and 27‑51× faster processing for log-to-metric and pack-fields respectively.

End‑to‑end tests using mock data sets and key module benchmarks confirm substantial gains in both latency and resource utilization.

Conclusion

The SPL iteration focuses on performance, scenario diversity, and usability, delivering a robust, cloud‑native solution for log aggregation, time‑series conversion, and observability, while laying a foundation for future enhancements.