BestHub
Sign in
Tagged articles

log storage

19 articles · Page 1 of 1
Cloud Architecture
Cloud Architecture
May 3, 2026 · Big Data

Cutting Log Storage Costs 70% for 100 Billion Daily Logs: A Full Guide to Hot‑Cold Separation Architecture

This article explains why massive log systems must adopt hot‑cold separation, walks through the problem analysis, SLO definition, component design, Kafka partition planning, Elasticsearch and ClickHouse tuning, Parquet archiving, a unified query gateway with async cold queries, governance practices, cost modeling, common pitfalls, and a roadmap for evolving the platform.

ElasticsearchKafkahot cold separation
0 likes · 40 min read
Cutting Log Storage Costs 70% for 100 Billion Daily Logs: A Full Guide to Hot‑Cold Separation Architecture
Ray's Galactic Tech
Ray's Galactic Tech
Jan 23, 2026 · Backend Development

How to Build a Kafka‑Level High‑Performance Message Queue from Scratch

This article presents a step‑by‑step guide to designing and implementing a Kafka‑class distributed log‑based message queue kernel, covering architecture, sequential writes, sparse indexing, zero‑copy I/O, partitioning, replication, consumer‑group metadata, batch pipelines, crash recovery, and performance benchmarks.

IndexingKafkaMessage Queue
0 likes · 7 min read
How to Build a Kafka‑Level High‑Performance Message Queue from Scratch
Ray's Galactic Tech
Ray's Galactic Tech
Nov 28, 2025 · Operations

How to Optimize Log Storage: From Centralized to Hot‑Cold Separation

This article explains why modern micro‑service systems need log storage optimization and presents a hot‑cold separation strategy, detailing ELK, Loki, and Kafka + ClickHouse architectures, implementation steps, best practices, and a comparative analysis to guide cost‑effective, high‑performance log management.

ClickHouseELKLoki
0 likes · 7 min read
How to Optimize Log Storage: From Centralized to Hot‑Cold Separation
Wukong Talks Architecture
Wukong Talks Architecture
Aug 21, 2025 · Operations

Why LinkedIn Dropped Kafka for Northguard – A Deep Dive into Its Architecture

LinkedIn, the creator of Kafka, has largely abandoned Kafka in favor of a new log storage system called Northguard, whose design mirrors Apache Pulsar with features like storage‑compute separation, log striping, and a multi‑layer data model, offering superior scalability, operability, consistency, and durability for massive data streams.

Apache PulsarLinkedInNorthguard
0 likes · 22 min read
Why LinkedIn Dropped Kafka for Northguard – A Deep Dive into Its Architecture
dbaplus Community
dbaplus Community
Mar 12, 2024 · Databases

How Didi Scaled Log Search by Replacing Elasticsearch with ClickHouse

Facing PB‑scale daily logs and costly Elasticsearch bottlenecks, Didi redesigned its log‑search architecture by migrating to ClickHouse, detailing the challenges, storage redesign, cluster upgrades, performance optimizations, stability fixes, and the resulting cost reduction and query speed gains.

ClickHousedistributed systemselasticsearch migration
0 likes · 15 min read
How Didi Scaled Log Search by Replacing Elasticsearch with ClickHouse
Linux Code Review Hub
Linux Code Review Hub
Mar 11, 2024 · Databases

How Didi Built a Next‑Gen Log Storage System with ClickHouse

Didi migrated its massive PB‑scale log data from Elasticsearch to ClickHouse, redesigning storage with separate Log and Trace clusters, optimizing partition and sorting keys, introducing native TCP connectors, and revamping HDFS cold‑hot separation, achieving up to four‑fold query speed gains and 30% lower hardware costs.

ClickHouseFlinkHDFS
0 likes · 15 min read
How Didi Built a Next‑Gen Log Storage System with ClickHouse
ITPUB
ITPUB
Oct 21, 2022 · Databases

How We Replaced Elasticsearch with ClickHouse for High‑Performance Log Storage

Facing rapid growth, our team evaluated ClickHouse’s hot‑cold storage and tiered‑disk policies to replace Elasticsearch, designing partitioning, TTL, and multi‑level storage strategies—including hot, cold, and archive disks, custom storage policies, and OSS integration—to achieve higher write throughput, better compression, and over 50% cost reduction.

ClickHouseCold Hot SeparationDatabase Migration
0 likes · 22 min read
How We Replaced Elasticsearch with ClickHouse for High‑Performance Log Storage
dbaplus Community
dbaplus Community
Oct 11, 2022 · Big Data

How We Replaced Elasticsearch with ClickHouse for Faster, Cheaper Log Storage

Facing growing log volumes and compliance needs, we evaluated ClickHouse’s hot‑cold‑archive storage to replace Elasticsearch, detailing configuration of storage policies, partitioning strategies, table creation, TTL handling, and cost‑effective OSS integration, ultimately achieving higher write performance and over 50% storage cost reduction.

Big DataClickHouseCold Hot Architecture
0 likes · 22 min read
How We Replaced Elasticsearch with ClickHouse for Faster, Cheaper Log Storage
Tencent Cloud Developer
Tencent Cloud Developer
Sep 26, 2022 · Big Data

Kafka Architecture Overview and Core Concepts

Kafka’s architecture consists of brokers forming clusters, producers publishing to topics split into partitions with replicas, consumers organized in groups pulling messages by offset, ZooKeeper managing metadata, and log‑based storage using append‑only files, indexes, and zero‑copy, while configurable acknowledgment, batching, and replication ensure high throughput and fault‑tolerant reliability.

KafkaProducerconsumer
0 likes · 18 min read
Kafka Architecture Overview and Core Concepts
Big Data Technology Architecture
Big Data Technology Architecture
Sep 17, 2022 · Databases

Design and Optimization of Bilibili Log Service 2.0 Using ClickHouse and OpenTelemetry

This article describes how Bilibili redesigned its log service by replacing Elasticsearch with ClickHouse, introducing OpenTelemetry‑based logging, optimizing storage, query, and alerting components, and enhancing ClickHouse features such as configuration tuning, Map types, and implicit columns to achieve higher performance, lower cost, and better observability.

ClickHousedatabase optimizationlog infrastructure
0 likes · 28 min read
Design and Optimization of Bilibili Log Service 2.0 Using ClickHouse and OpenTelemetry
Bilibili Tech
Bilibili Tech
Sep 16, 2022 · Big Data

Design and Optimization of Bilibili Log Service 2.0 Using ClickHouse and OpenTelemetry

Bilibili’s Log Service 2.0 replaces its Elastic‑Stack pipeline with an OpenTelemetry‑driven architecture that writes logs via high‑performance Go/Java SDKs to ClickHouse, delivering ten‑fold write throughput, two‑fold query speed, one‑third storage cost, a custom query gateway, visualization UI, and advanced alerting.

ClickHouselog infrastructurelog storage
0 likes · 27 min read
Design and Optimization of Bilibili Log Service 2.0 Using ClickHouse and OpenTelemetry
Java Architect Essentials
Java Architect Essentials
Jul 27, 2021 · Backend Development

Kafka Overview: Architecture, Core Features, and Operational Details

This article provides a comprehensive technical overview of Apache Kafka, covering its distributed messaging architecture, key features such as high‑throughput read/write, replication, partitioning, consumer group mechanics, offset management, rebalance processes, and practical code examples for synchronous and asynchronous offset commits.

Consumer OffsetsDistributed MessagingKafka
0 likes · 22 min read
Kafka Overview: Architecture, Core Features, and Operational Details
dbaplus Community
dbaplus Community
Jul 8, 2021 · Databases

Why ClickHouse Outperforms Elasticsearch for Log Storage and Analytics

This article compares ClickHouse and Elasticsearch for API log storage, detailing development activity, schema handling, query performance, statistical functions, MySQL integration, new features, and practical drawbacks, while providing concrete SQL examples and migration tips.

ClickHouseElasticsearchSQL
0 likes · 14 min read
Why ClickHouse Outperforms Elasticsearch for Log Storage and Analytics
Qunar Tech Salon
Qunar Tech Salon
Apr 30, 2016 · Big Data

Designing and Optimizing Log Storage and Query in HBase

This article analyzes the characteristics of log data, explains why HBase is chosen for log storage, discusses the shortcomings of self‑built indexes, and presents optimization strategies such as rowKey design, filter usage, coprocessor integration, and third‑party indexing to improve query performance.

HBaseIndexingQuery Optimization
0 likes · 12 min read
Designing and Optimizing Log Storage and Query in HBase
21CTO
21CTO
Apr 16, 2016 · Databases

Optimizing HBase Log Queries: Index Design and RowKey Strategies

This article examines the challenges of storing and querying log data in HBase, outlines the drawbacks of custom indexing, and presents practical rowKey design, filter usage, and integration with external search engines to improve query performance.

Big DataHBaseNoSQL
0 likes · 15 min read
Optimizing HBase Log Queries: Index Design and RowKey Strategies