Why Observability 2.0 and SLS Data Pipelines Are Revolutionizing Log Analytics

Observability 2.0 Overview

Observability 2.0 (o11y 2.0) has become a hot topic in DevOps, highlighted by Honeycomb’s "Introducing Observability 2.0" and CNCF’s "What is observability 2.0?". It aims to break the siloed Log/Metric/Trace approach, provide a unified health view, accelerate system behavior understanding, and apply AI for anomaly detection and fault localization.

Key Challenges Addressed by o11y 2.0

Unify logs, metrics and traces on a single platform to create a complete system health view.

Enable engineers to diagnose issues faster, reducing downtime.

Make logs the core data source, enrich them with wide‑event context, and reconstruct facts.

Leverage cloud‑native elasticity for massive event querying and analysis.

Evolution of the Observability Data Pipeline

SLS (Alibaba Cloud Log Service) introduced data processing in 2019 and began a major pipeline upgrade in 2024, delivering four improvements described as "more, faster, better, cheaper".

The upgraded pipeline offers three service modes:

All three modes address different scenarios, fault tolerance, cost and ecosystem integration requirements.

Performance Improvements

SLS now uses the SPL engine (columnar computation, SIMD acceleration, C++ implementation) to achieve millisecond‑level latency for log ETL. In burst scenarios, the new pipeline matches log generation speed and keeps delay under one second, while handling up to 1 PB/day of raw logs.

Cost Savings

The new pipeline reduces processing costs by 66.7% compared with the legacy Python‑DSL engine, lowering total cost of ownership beyond compute fees. Savings also come from reduced storage‑fragment fees and simplified resource management.

Integration Efficiency

SLS integrates with Flink, Spark, Flume, DataWorks, OSS, and other services. Using SPL low‑code, common filtering and field extraction can be pushed to SLS, cutting Python development time and improving CPU efficiency. Example SPL statements:

* | project time_local, request_uri, status, user_agent, client_ip

* | where status != '200'

In a high‑filter scenario, processing 10 MB of data dropped from 10‑15 seconds to 300 ms, dramatically reducing latency and function‑as‑a‑service costs.

Network Cost Optimization

Cross‑region bandwidth dominates pipeline costs. Strategies include compressing data streams (e.g., ZSTD) and transmitting only necessary columns or rows. SPL can perform column projection and row filtering directly at the source, cutting traffic.

Conclusion

Observability 2.0 and the SPL‑driven SLS pipeline provide schema‑free processing, wide‑event enrichment, real‑time high performance, and flexible scalability. Ongoing enhancements will further strengthen these capabilities for AI‑driven observability workloads.