Tagged articles
Log Storage
20 articles · Page 1 of 1
This article details a production-grade Spring Boot audit architecture covering full-chain traceability, async log persistence, dynamic data masking via Jackson serializers, tamper-proofing with hash chains, and hot/cold log tiering, with concrete code examples and performance tuning insights from financial and government deployments.
ComplianceJacksonLog Storage
0 likes · 22 min read

This article explains why massive log systems must adopt hot‑cold separation, walks through the problem analysis, SLO definition, component design, Kafka partition planning, Elasticsearch and ClickHouse tuning, Parquet archiving, a unified query gateway with async cold queries, governance practices, cost modeling, common pitfalls, and a roadmap for evolving the platform.
ElasticsearchKafkaLog Storage
0 likes · 40 min read

This article presents a step‑by‑step guide to designing and implementing a Kafka‑class distributed log‑based message queue kernel, covering architecture, sequential writes, sparse indexing, zero‑copy I/O, partitioning, replication, consumer‑group metadata, batch pipelines, crash recovery, and performance benchmarks.
IndexingKafkaLog Storage
0 likes · 7 min read

This article explains why modern micro‑service systems need log storage optimization and presents a hot‑cold separation strategy, detailing ELK, Loki, and Kafka + ClickHouse architectures, implementation steps, best practices, and a comparative analysis to guide cost‑effective, high‑performance log management.
ClickHouseLog StorageLoki
0 likes · 7 min read

LinkedIn, the creator of Kafka, has largely abandoned Kafka in favor of a new log storage system called Northguard, whose design mirrors Apache Pulsar with features like storage‑compute separation, log striping, and a multi‑layer data model, offering superior scalability, operability, consistency, and durability for massive data streams.
Apache PulsarLinkedInLog Storage
0 likes · 22 min read

Facing PB‑scale daily logs and costly Elasticsearch bottlenecks, Didi redesigned its log‑search architecture by migrating to ClickHouse, detailing the challenges, storage redesign, cluster upgrades, performance optimizations, stability fixes, and the resulting cost reduction and query speed gains.
ClickHouseLog Storagedistributed systems
0 likes · 15 min read

Didi migrated its massive PB‑scale log data from Elasticsearch to ClickHouse, redesigning storage with separate Log and Trace clusters, optimizing partition and sorting keys, introducing native TCP connectors, and revamping HDFS cold‑hot separation, achieving up to four‑fold query speed gains and 30% lower hardware costs.
ClickHouseFlinkHDFS
0 likes · 15 min read

This article summarizes practical experiences with ClickHouse log storage, covering how to size and tune clusters, key table schema design considerations, partitioning strategies, index choices, compression algorithms, and provides a demo CREATE TABLE script for production use.
ClickHouseCluster ScalingCompression
0 likes · 11 min read

This article thoroughly examines Kafka's storage system, explaining why it uses sequential log writes combined with sparse indexing, how different log formats evolved, and the mechanisms for log retention and compaction that enable high‑throughput, fault‑tolerant streaming at massive scale.
KafkaLog CompactionLog Retention
0 likes · 22 min read


ITPUBOct 21, 2022 · Databases
Facing rapid growth, our team evaluated ClickHouse’s hot‑cold storage and tiered‑disk policies to replace Elasticsearch, designing partitioning, TTL, and multi‑level storage strategies—including hot, cold, and archive disks, custom storage policies, and OSS integration—to achieve higher write throughput, better compression, and over 50% cost reduction.
ClickHouseCold Hot SeparationCost Optimization
0 likes · 22 min read

Facing growing log volumes and compliance needs, we evaluated ClickHouse’s hot‑cold‑archive storage to replace Elasticsearch, detailing configuration of storage policies, partitioning strategies, table creation, TTL handling, and cost‑effective OSS integration, ultimately achieving higher write performance and over 50% storage cost reduction.
ClickHouseCold Hot ArchitectureLog Storage
0 likes · 22 min read

To replace Elasticsearch, the team built a ClickHouse‑based log platform that uses table‑level TTL and custom storage policies to automatically move data among hot (ESSD PL1), cold (ESSD PL0) and archive (OSS) disks, achieving faster writes, comparable query latency, and over 50 % cost reduction.
ClickHouseHot-Cold TieringLog Storage
0 likes · 21 min read

Kafka’s architecture consists of brokers forming clusters, producers publishing to topics split into partitions with replicas, consumers organized in groups pulling messages by offset, ZooKeeper managing metadata, and log‑based storage using append‑only files, indexes, and zero‑copy, while configurable acknowledgment, batching, and replication ensure high throughput and fault‑tolerant reliability.
ConsumerKafkaLog Storage
0 likes · 18 min read

This article describes how Bilibili redesigned its log service by replacing Elasticsearch with ClickHouse, introducing OpenTelemetry‑based logging, optimizing storage, query, and alerting components, and enhancing ClickHouse features such as configuration tuning, Map types, and implicit columns to achieve higher performance, lower cost, and better observability.
ClickHouseLog StorageOpenTelemetry
0 likes · 28 min read

Bilibili’s Log Service 2.0 replaces its Elastic‑Stack pipeline with an OpenTelemetry‑driven architecture that writes logs via high‑performance Go/Java SDKs to ClickHouse, delivering ten‑fold write throughput, two‑fold query speed, one‑third storage cost, a custom query gateway, visualization UI, and advanced alerting.
ClickHouseLog StorageOpenTelemetry
0 likes · 27 min read

This article provides a comprehensive technical overview of Apache Kafka, covering its distributed messaging architecture, key features such as high‑throughput read/write, replication, partitioning, consumer group mechanics, offset management, rebalance processes, and practical code examples for synchronous and asynchronous offset commits.
Consumer OffsetsDistributed MessagingKafka
0 likes · 22 min read

This article compares ClickHouse and Elasticsearch for API log storage, detailing development activity, schema handling, query performance, statistical functions, MySQL integration, new features, and practical drawbacks, while providing concrete SQL examples and migration tips.
AnalyticsClickHouseElasticsearch
0 likes · 14 min read

The article compares ClickHouse, Elasticsearch, and MySQL for log storage and analysis, highlighting ClickHouse's active development, SQL support, JSON handling, performance, storage efficiency, integration options, and practical query examples to help developers decide the best backend for observability pipelines.
AnalyticsJSONLog Storage
0 likes · 12 min read

This article analyzes the characteristics of log data, explains why HBase is chosen for log storage, discusses the shortcomings of self‑built indexes, and presents optimization strategies such as rowKey design, filter usage, coprocessor integration, and third‑party indexing to improve query performance.
HBaseIndexingLog Storage
0 likes · 12 min read


21CTOApr 16, 2016 · Databases
This article examines the challenges of storing and querying log data in HBase, outlines the drawbacks of custom indexing, and presents practical rowKey design, filter usage, and integration with external search engines to improve query performance.
HBaseLog StorageNoSQL
0 likes · 15 min read
