Design and Implementation of a Scalable Real-Time Log Monitoring Platform at Baidu

This article introduces Baidu's log platform that handles billions of daily events, explains UBC logging concepts and monitoring requirements, and details a low‑cost, high‑accuracy architecture using real‑time streaming, dimension mapping, watermarking, and time‑window aggregation to achieve reliable, scalable event monitoring.

Big DataLog MonitoringReal-time Streaming

0 likes · 14 min read

Design and Implementation of a Scalable Real-Time Log Monitoring Platform at Baidu

Architect

Aug 7, 2024 · Operations

How Baidu Scales Real‑Time Log Monitoring for Billions of Events

This article explains Baidu's log‑center architecture for handling billions of UBC events per day, detailing UBC concepts, monitoring requirements, a low‑cost scalable design with dimension mapping, watermarking, data trimming and time‑window aggregation, and the resulting performance and cost benefits.

Cost OptimizationLog MonitoringReal-time Streaming

0 likes · 13 min read

How Baidu Scales Real‑Time Log Monitoring for Billions of Events

Baidu Tech Salon

Jun 18, 2024 · Big Data

Scalable, High‑Accuracy Event Logging Monitoring for Baidu's Log Platform

Baidu’s log platform processes billions of daily page‑view events and, to monitor them accurately with minute‑level latency, implements a downstream streaming‑task architecture that maps limited custom dimensions, uses watermarks for completeness, trims raw data, aggregates into 5‑minute windows, and outputs concise metrics to Elasticsearch, achieving high accuracy, configurability, and low cost.

Log MonitoringReal-time analyticsUBC

0 likes · 11 min read

Scalable, High‑Accuracy Event Logging Monitoring for Baidu's Log Platform

Baidu Geek Talk

Jun 17, 2024 · Industry Insights

How Baidu Scales Real‑Time Event Monitoring for Billions of Log Events

This article explains Baidu's log platform architecture, the UBC event‑tracking protocol, monitoring requirements, and the low‑cost, high‑accuracy solutions—including dimension mapping, watermark handling, data trimming, and time‑window aggregation—that enable real‑time, customizable monitoring of petabyte‑scale log streams.

Cost OptimizationLog MonitoringReal-time analytics

0 likes · 13 min read

How Baidu Scales Real‑Time Event Monitoring for Billions of Log Events