Tagged articles
StreamSQL
11 articles · Page 1 of 1
To support Didi’s fast‑growing car‑pool service, a real‑time data warehouse was built using a streamlined layered architecture—ODS, DWD, DIM, DWM, and APP—leveraging Flink‑based StreamSQL, Kafka, Druid and ClickHouse to deliver minute‑level analytics, dashboards, monitoring, and cross‑business interfaces while planning unified meta‑store integration.
FlinkStreamSQLbig data architecture
0 likes · 20 min read

Didi has unified its real‑time computing on Apache Flink, creating an enhanced StreamSQL service with extended DDL, built‑in parsers and UDX, supporting thousands of nodes, millions of jobs, and trillions of daily records, while addressing state management, high availability, multi‑language UDFs, and pursuing real‑time ML and data‑warehouse integration.
Apache FlinkDidiReal-time Computing
0 likes · 13 min read

Senior Didi technology expert Liang Li-yin shares how Didi leverages Apache Flink for large‑scale real‑time computing, covering service architecture, StreamSQL advantages, multi‑cluster management, task control, monitoring, meta‑store integration, challenges, and future plans such as high availability, real‑time ML, and unified batch‑stream processing.
Apache FlinkData EngineeringReal-time Computing
0 likes · 14 min read

This article describes how Didi adopted Apache Flink for its real‑time data streams, detailing the platformized architecture, production use cases such as ETL, monitoring and CEP, the evolution of StreamSQL, and the engineering improvements made to support large‑scale, low‑latency processing.
DidiFlinkStreamSQL
0 likes · 14 min read

This article outlines Didi's journey from fragmented, self‑built real‑time clusters to a unified, YARN‑managed platform that leverages Spark Streaming, Flink, and StreamSQL, detailing architectural choices, resource isolation, CEP enhancements, and the resulting impact on latency‑critical services.
CEPFlinkSpark Streaming
0 likes · 10 min read

From early self‑built Storm and Spark Streaming clusters to a unified YARN‑based Spark platform and finally a low‑latency Flink system with extended CEP and StreamSQL capabilities, Didi’s real‑time computing platform evolved through three stages, delivering multi‑tenant isolation, rich SQL processing, and dramatically reduced development costs.
CEPFlinkReal-time Computing
0 likes · 9 min read

Transwarp Slipstream’s Complex Event Processing (CEP) engine enables developers to define and detect sophisticated event patterns using SQL, offering detailed syntax, configuration steps, and practical use cases such as bank fraud detection and traffic monitoring, thereby enhancing real‑time analytics in big‑data environments.
CEPStreamSQL
0 likes · 9 min read

This article explains how StreamSQL runs on the Slipstream engine in event‑driven mode, shows how to enable the mode, and provides step‑by‑step code examples for low‑latency stream processing, window aggregation, and joining multiple window streams.
StreamSQLWindow Aggregationbig data
0 likes · 9 min read

This article explains the StreamSQL Application concept, its resource‑sharing and isolation mechanisms, the DDL syntax for managing Applications, and a step‑by‑step example showing how a new user can create, run, and isolate StreamJobs without affecting existing production workloads.
StreamSQLapplicationresource isolation
0 likes · 8 min read

This article introduces Inceptor StreamSQL, explains its core concepts of Stream, StreamJob, and Application, and provides a step‑by‑step tutorial—from creating a Kafka source to launching a StreamJob and querying results—highlighting its ease of use and performance benefits.
InceptorKafkaStreamSQL
0 likes · 10 min read

This article compares Lambda and Kappa architectures, explains their three‑layer models, highlights the drawbacks of maintaining separate batch and speed layers in Lambda, introduces Kappa’s unified approach with StreamSQL, provides a smart‑traffic case study, and offers guidance on choosing the right architecture based on data volume, development complexity, and operational costs.
Kappa architectureLambda architectureStreamSQL
0 likes · 15 min read
