0 views collected around this technical thread.
To import one billion 1 KB log records stored in HDFS or S3 into MySQL efficiently, the article examines data partitioning, B‑tree index limits, batch insertion, storage engine choices, concurrency control, file‑reading methods, task scheduling with Redis, Redisson, and Zookeeper for reliable, ordered, high‑throughput loading.
This article explains how to implement real‑time materialized views to accelerate large‑scale time‑series data queries, covering the need for materialized views, their definition, storage, incremental updates, pre‑computation, query partitioning, performance testing, and future directions.
The article explains how to handle billions of daily orders by classifying data into hot and cold segments, storing them in MySQL, Elasticsearch, and Hive, and applying sharding and partitioning techniques at both table and database levels to achieve scalable performance.
This article explains the fundamental concepts of Redis cluster architecture, covering high‑availability with Sentinel, data partitioning methods, proxy‑based sharding techniques, the mechanics of Redis Cluster without a central node, and practical considerations for multi‑key operations in a distributed environment.
This article chronicles the architectural evolution of JD Baitiao’s backend—from early MySQL monoliths through Solr‑HBase, MongoDB, and DBRep—to the adoption of Apache ShardingSphere, highlighting the motivations, technical trade‑offs, decoupling strategies, and performance outcomes for a high‑throughput financial service.
In distributed databases, proxies must merge sorted results from multiple shards, but large result sets exceed memory limits; the article proposes a batch‑fetching approach using per‑shard sort buffers and a priority‑queue merge, eliminating disk I/O and reducing network waste while preserving global order.
Effective MySQL sharding requires sustainable, low‑skew designs, favoring hash‑based methods with proper coprime counts, two‑stage partitioning, routing tables, or consistent hashing, while supporting expansion via doubling or flexible consistent‑hash growth to avoid hot spots and uneven data distribution.
This article walks through the challenges of querying a 300‑billion‑row Hive table, analyzes why traditional partitioning, indexing, and bucketing fall short, and presents a practical solution that combines active‑user segmentation and a redesigned array‑based data model to cut query time from hours to minutes.
This article explains how ClickHouse's MergeTree engine creates partition directories based on a partition key, details the naming convention PartitionID_MinBlockNum_MaxBlockNum_Level, and describes the automatic and manual merge processes that consolidate partitions for efficient storage.
This article explains the principles of MySQL performance optimization, distinguishes active and passive approaches, and provides concrete solutions for slow single queries, partially slow queries, and overall slow queries through proper indexing, data partitioning, slow‑query‑log configuration, and read‑write splitting.
This article explains the primary use cases for distributed databases, contrasts them with traditional databases, and describes how data partitioning and metadata enable clients to locate data without scanning all nodes, highlighting both external user benefits and internal implementation challenges.
The article introduces the Cube Model for scalable microservice architectures, explaining how X‑axis (horizontal scaling), Y‑axis (functional decomposition) and Z‑axis (data partitioning and isolation) expansions address capacity, complexity, and differentiated service demands in high‑traffic distributed systems.
Database sharding, a horizontal partitioning technique that splits a table’s rows across multiple nodes, enables scalable performance and fault isolation for high‑traffic applications, but introduces complexity, potential data imbalance, and recovery challenges, so it should be adopted only after simpler optimizations are exhausted.
This article presents a comprehensive case study of how WeChat Pay rebuilt its transaction record storage system to handle massive data volumes, improve performance, ensure data completeness, support flexible queries, and strengthen security through distributed key‑value storage, data partitioning, and operational safeguards.
The article presents a comprehensive case study of how WeChat Pay rebuilt its transaction record storage to handle massive data growth, improve performance, ensure data completeness, support flexible queries, and strengthen security through distributed key‑value storage, hierarchical partitioning, holiday traffic shaping, and strict access controls.