0 views collected around this technical thread.
This article details CVTE’s adoption of stream computing, describing the company background, the challenges of traditional data pipelines, the design of a CDC‑Kafka integration platform, evaluations of PipelineDB, ksqlDB, Materialize and RisingWave, and the overall impact on real‑time analytics and operational efficiency.
Baidu’s logging platform centralizes data collection, transmission, management, and analysis for billions of daily logs, employing a layered architecture with priority persistence, service decomposition, stream computing, and client‑side optimizations to guarantee no duplication, no loss, and 99.99%+ stability.
Tencent Cloud Oceanus, a computing service powering internal apps like WeChat and external partners such as Bilibili, scales to over 30,000 cores handling 5 PB daily and 500,000 jobs, and tackles Flink SQL’s syntax, function and operational limits with table‑valued functions, incremental and enhanced tumble windows, and caching‑based retraction optimization that cuts downstream data volume up to 30× and improves join performance by about 20 %.
BehaviR is a mobile‑side real‑time stream processing framework that gathers page actions, network requests, and storage events via JSON configuration, runs stateful computations on a lightweight Python‑based engine using a DSL, and instantly triggers decisions and routes results with millisecond latency and no server load.
As mobile and IoT data surge, real-time stream computing—especially Flink’s low-latency, high-throughput, exactly-once engine—addresses challenges of latency, accuracy, and usability, and Tencent Cloud’s managed Flink service provides elastic, secure, integrated pipelines for applications ranging from online status monitoring to fraud detection and smart transportation.
This article outlines Sogou's infrastructure evolution under rapid business iteration, detailing stages of compute and storage horizontal scaling, serviceization, and stream computing, while sharing the practices, principles, lessons learned, and reflections that guided the platform's architectural transformation.