Tagged articles
log platform
17 articles · Page 1 of 1
Starbucks China’s logging team migrated several petabytes of logs from legacy VM‑based Elasticsearch clusters to a cloud‑native bare‑metal Kubernetes platform, upgrading ES from 7.x to 8.x, containerizing components, optimizing storage and Kafka, and achieving up to 80% query speed gains, 30% CPU reduction, and 200% write‑throughput improvement.
Performance Optimizationdata pipelinelog platform
0 likes · 25 min read

This article explains how to build a log platform by integrating native Kibana with ClickHouse using an open‑source proxy (CKibana), covering migration motivations, architecture, configuration steps, advanced features like sampling and caching, and the resulting cost and stability benefits.
ClickHouseKibanabig data
0 likes · 12 min read

The Venus log platform at iQiyi migrated from an ElasticSearch‑Kibana architecture to an Iceberg‑based data lake with Trino, cutting storage and compute costs by over 70%, boosting stability by 85%, and efficiently supporting billions of daily logs through write‑heavy, low‑query workloads.
ElasticsearchIcebergarchitecture evolution
0 likes · 22 min read

This article details Bilibili's evolution from the ClickHouse‑based Billions 2.0 log system to the Billions 3.0 architecture, explaining how they reduced storage costs, improved troubleshooting, adopted a lake‑house design with Iceberg on HDFS, leveraged ClickHouse for acceleration, and integrated Trino as the unified query engine.
ClickHouseIceberglog platform
0 likes · 37 min read

Bilibili evolved its log platform from ClickHouse‑based Billions 2.0 to Billions 3.0 lakehouse using Iceberg, HDFS, Trino, retaining ClickHouse for acceleration; this reduces storage cost by over 20%, improves observability, solves the compute‑storage mismatch, adds flexible indexing, and supports complex ETL while staying open‑source.
ClickHouseIcebergindexing
0 likes · 36 min read

This article details the background, challenges, migration plan, automation, and performance‑tuning techniques used to relocate a petabyte‑scale Elasticsearch logging cluster from one data center to another, highlighting practical lessons and measurable improvements in stability and migration speed.
AutomationCluster MigrationElasticsearch
0 likes · 20 min read

NetEase Yanxuan built a unified log platform that consolidates collection, delivery, processing, storage, analysis, and alerting for near‑real‑time and offline logs, using Flume, a Golang‑based Loggie agent, Kafka, Flink, HBase and Elasticsearch to ensure high performance, data‑quality, container‑native isolation, full‑link traceability, and automated scaling, with Loggie now open‑sourced.
Log Collectionlog managementlog platform
0 likes · 38 min read

This article explains the design, implementation, and future plans of Baidu's log middle platform, detailing its lifecycle management, service architecture, data reliability goals of eliminating duplication and loss, and the technical measures taken across SDKs, servers, and streaming pipelines to achieve near‑100% data integrity.
Backend ArchitectureData Reliabilitybig data
0 likes · 15 min read

This article describes Baidu's log middle platform architecture, its data lifecycle management, integration status, terminology, service overview, core challenges of ensuring data accuracy, and the implemented optimizations for persistent storage, service decomposition, and SDK reporting to achieve near‑100% no‑repeat no‑loss reliability.
Backend ArchitectureData Reliabilitybig data
0 likes · 15 min read

This article introduces the current state of Baidu's log platform, explains its lifecycle from data collection to downstream applications, analyzes the challenges of achieving near‑zero duplication and loss, and presents architectural optimizations and best‑practice recommendations to improve data stability and accuracy across the system.
Data Reliabilitybig datadata pipeline
0 likes · 19 min read

Baidu’s logging platform centralizes data collection, transmission, management, and analysis for billions of daily logs, employing a layered architecture with priority persistence, service decomposition, stream computing, and client‑side optimizations to guarantee no duplication, no loss, and 99.99%+ stability.
Architecture DesignBaiduData Accuracy
0 likes · 11 min read

This article details the design and implementation of a unified log platform that handles peak write rates of up to one million events per second, balances performance, stability, and cost, and leverages Filebeat, Kafka, Flink, and Elasticsearch across multi‑cloud environments.
ElasticsearchKafkalog platform
0 likes · 13 min read

This article details JD's end‑to‑end seller log system architecture, explaining why Kafka, Storm, Elasticsearch and HBase were chosen, the challenges faced during scaling, and the practical solutions implemented to achieve a unified, high‑throughput logging platform for merchants and operations.
ElasticsearchHBaseKafka
0 likes · 13 min read

The article describes how Airbnb's data infrastructure team built a next‑generation log storage and query platform to improve data quality, timeliness, flexibility, and anomaly detection, outlining the system architecture, key requirements, five improvement areas, and the resulting benefits.
Airbnbdata pipelinelog platform
0 likes · 7 min read

The article details the design, components, and operational challenges of Youzan's unified log platform, describing its multi‑layer architecture, ingestion methods using rsyslog/logstash and Flume‑NG, Kafka‑based log center, processing pipelines with Storm/Spark, and storage in HDFS and Elasticsearch.
Distributed SystemsFlumeKafka
0 likes · 10 min read

The article describes the design, components, and implementation details of Youzan's unified log platform, covering log ingestion via rsyslog, Logstash, and Flume, centralized processing with Kafka, real‑time analysis using Storm/Spark, and storage in HDFS, Elasticsearch, and Hawk, while also discussing challenges and future improvements.
ElasticsearchHDFSKafka
0 likes · 10 min read

This article describes how Qunar's DevOps team built a Mesos‑powered log analysis platform using ELK, Marathon, Chronos, and custom monitoring frameworks, detailing deployment architecture, resource management, container logging, image handling, and operational lessons learned.
ChronosDevOpsMarathon
0 likes · 15 min read
