Tagged articles
Data Partitioning
29 articles · Page 1 of 1
This article analyzes the challenges of JD.com's POP order Elasticsearch storage—including data skew, oversized shards, frequent updates, and high maintenance costs—and details the multi‑layered architectural redesign that introduced tenant isolation, dual‑hash routing, differentiated shard strategies, and a dual‑active physical foundation to achieve high performance, scalability, and availability.
Data PartitioningElasticsearchorder management
0 likes · 16 min read

This article examines ClickHouse’s high‑performance OLAP design, covering its MPP architecture, columnar storage, vectorized execution, pre‑sorting, table engines, extensive data‑type system, sharding and replication strategies, as well as its sparse and skip‑index mechanisms that together enable ultra‑fast analytics on massive datasets.
ClickHouseData PartitioningIndexing
0 likes · 16 min read

To import one billion 1 KB log records stored in HDFS or S3 into MySQL efficiently, the article examines data partitioning, B‑tree index limits, batch insertion, storage engine choices, concurrency control, file‑reading methods, task scheduling with Redis, Redisson, and Zookeeper for reliable, ordered, high‑throughput loading.
ConcurrencyData PartitioningMySQL
0 likes · 18 min read

This article explains how to implement real‑time materialized views to accelerate large‑scale time‑series data queries, covering the need for materialized views, their definition, storage, incremental updates, pre‑computation, query partitioning, performance testing, and future directions.
Data PartitioningMaterialized ViewQuery Acceleration
0 likes · 16 min read


ITPUBApr 27, 2023 · Databases
This guide walks through diagnosing a MongoDB outage, explains sharding concepts, shows when to shard, and provides step‑by‑step instructions for configuring config servers, mongos routers, converting replica sets, sharding collections, tracking chunks, and managing the balancer.
BalancerData PartitioningDatabase Cluster
0 likes · 13 min read


ITPUBApr 1, 2023 · Databases
Learn why large-scale internet applications need database sharding, understand the evolution from single databases to split databases and tables, and follow practical steps for assessing, planning, and implementing horizontal and vertical sharding, including sizing calculations, key selection, and query challenges.
Data PartitioningHorizontal ShardingVertical Sharding
0 likes · 14 min read

This article details GitHub’s 12‑step journey from a 12‑year‑old Ruby on Rails monolith to a modular microservices architecture, covering rapid growth, data partitioning, service extraction, operational shifts, and resilient design principles to empower developers while maintaining product value.
Data PartitioningGitHubbackend architecture
0 likes · 16 min read

The article explains how to handle billions of daily orders by classifying data into hot and cold segments, storing them in MySQL, Elasticsearch, and Hive, and applying sharding and partitioning techniques at both table and database levels to achieve scalable performance.
Data PartitioningElasticsearchHive
0 likes · 9 min read


ITPUBDec 29, 2022 · Databases
This transcript details the design, implementation, and performance evaluation of TDSQL‑C's parallel query feature, covering product background, workload challenges, execution plans, Amdahl's law analysis, task scheduling, data partitioning, and future enhancements for cloud‑native database systems.
Amdahl's LawCloud-Native DatabaseData Partitioning
0 likes · 17 min read

This article explains the fundamental concepts of Redis cluster architecture, covering high‑availability with Sentinel, data partitioning methods, proxy‑based sharding techniques, the mechanics of Redis Cluster without a central node, and practical considerations for multi‑key operations in a distributed environment.
ClusterConsistent HashingData Partitioning
0 likes · 9 min read

When service load spikes, instead of merely adding machines, this article explains how the AKF Scale Cube model—covering X‑axis horizontal scaling, Y‑axis functional or business splitting, and Z‑axis data partitioning—offers elegant, fine‑grained strategies to boost microservice performance and reliability.
AKF Scale CubeData Partitioningfunctional splitting
0 likes · 10 min read


ITPUBMay 27, 2022 · Databases
This article explains the fundamentals of graph computing, compares it with traditional processing, outlines industry challenges such as partitioning and load imbalance, and details HugeGraph’s self‑developed architecture, key technical solutions, and how developers can create and deploy graph algorithms.
Algorithm DevelopmentData PartitioningHugeGraph
0 likes · 14 min read

This article chronicles the architectural evolution of JD Baitiao’s backend—from early MySQL monoliths through Solr‑HBase, MongoDB, and DBRep—to the adoption of Apache ShardingSphere, highlighting the motivations, technical trade‑offs, decoupling strategies, and performance outcomes for a high‑throughput financial service.
Data PartitioningDatabase ArchitectureJD Baitiao
0 likes · 15 min read

The article explains how to address single‑point failures, limited capacity, and performance bottlenecks when scaling a single‑node service by introducing the AKF microservice design principle—X‑axis horizontal replication, Y‑axis functional separation, and Z‑axis data‑driven partitioning—detailing each split type with diagrams and practical considerations.
AKF principleData PartitioningHorizontal Scaling
0 likes · 5 min read

In distributed databases, proxies must merge sorted results from multiple shards, but large result sets exceed memory limits; the article proposes a batch‑fetching approach using per‑shard sort buffers and a priority‑queue merge, eliminating disk I/O and reducing network waste while preserving global order.
Data PartitioningDatabase ArchitectureSorting Algorithms
0 likes · 15 min read

Effective MySQL sharding requires sustainable, low‑skew designs, favoring hash‑based methods with proper coprime counts, two‑stage partitioning, routing tables, or consistent hashing, while supporting expansion via doubling or flexible consistent‑hash growth to avoid hot spots and uneven data distribution.
Consistent HashingData PartitioningHash Sharding
0 likes · 23 min read

This article explains the AKF scalability cube for microservice architecture, detailing the Y‑axis functional decomposition, X‑axis horizontal scaling, Z‑axis data partitioning, as well as front‑end/back‑end separation, stateless services, and RESTful communication, providing practical diagrams and guidance for building robust, scalable systems.
Data PartitioningRESTfulStateless Services
0 likes · 9 min read

This article walks through the challenges of querying a 300‑billion‑row Hive table, analyzes why traditional partitioning, indexing, and bucketing fall short, and presents a practical solution that combines active‑user segmentation and a redesigned array‑based data model to cut query time from hours to minutes.
Data ModelingData PartitioningHive
0 likes · 10 min read

This article explains how ClickHouse's MergeTree engine creates partition directories based on a partition key, details the naming convention PartitionID_MinBlockNum_MaxBlockNum_Level, and describes the automatic and manual merge processes that consolidate partitions for efficient storage.
ClickHouseData PartitioningMergeTree
0 likes · 8 min read

This article explains the principles of MySQL performance optimization, distinguishes active and passive approaches, and provides concrete solutions for slow single queries, partially slow queries, and overall slow queries through proper indexing, data partitioning, slow‑query‑log configuration, and read‑write splitting.
Data PartitioningMySQLperformance optimization
0 likes · 12 min read

The article reviews Google’s F1 Query paper, describing its architecture, three execution modes, data source handling, extensibility features such as UDF servers and TVFs, and performance optimizations that enable a unified, enterprise‑wide SQL engine for heterogeneous big‑data workloads.
Data PartitioningF1 QuerySQL
0 likes · 23 min read

This article explains the primary use cases for distributed databases, contrasts them with traditional databases, and describes how data partitioning and metadata enable clients to locate data without scanning all nodes, highlighting both external user benefits and internal implementation challenges.
Data PartitioningHigh Availabilitydistributed databases
0 likes · 3 min read

The article introduces the Cube Model for scalable microservice architectures, explaining how X‑axis (horizontal scaling), Y‑axis (functional decomposition) and Z‑axis (data partitioning and isolation) expansions address capacity, complexity, and differentiated service demands in high‑traffic distributed systems.
Data Partitioningbackend architectureload-balancing
0 likes · 9 min read

This article explains why and when to shard databases, compares sharding methods such as key‑based, range‑based, and dictionary approaches, and outlines the performance gains, availability improvements, and new challenges like increased complexity, ID handling, cross‑shard queries, and distributed transactions.
Data Partitioningdatabase shardingperformance
0 likes · 5 min read

Database sharding, a horizontal partitioning technique that splits a table’s rows across multiple nodes, enables scalable performance and fault isolation for high‑traffic applications, but introduces complexity, potential data imbalance, and recovery challenges, so it should be adopted only after simpler optimizations are exhausted.
Data PartitioningHorizontal Scalingdatabase sharding
0 likes · 15 min read

This article presents a comprehensive case study of how WeChat Pay rebuilt its transaction record storage system to handle massive data volumes, improve performance, ensure data completeness, support flexible queries, and strengthen security through distributed key‑value storage, data partitioning, and operational safeguards.
Data PartitioningSecurityWeChat Pay
0 likes · 11 min read


21CTODec 5, 2017 · Backend Development
This article details the challenges of WeChat Pay’s original transaction‑record system, the architectural redesign using distributed key‑value storage and data partitioning, and the resulting improvements in scalability, reliability, security, and user experience.
Data PartitioningDistributed KVWeChat Pay
0 likes · 11 min read


ITPUBNov 20, 2017 · Databases
This article examines the challenges of traditional database sharding, introduces core business object‑based partitioning, and details the design, architecture, and operation of the mysqlda MySQL middleware, including its benefits, limitations, and a practical deployment example.
Data PartitioningDatabase MiddlewareMySQL
0 likes · 12 min read


21CTOFeb 4, 2016 · Backend Development
This article outlines essential design principles for large‑scale web architectures—including availability, performance, reliability, scalability, manageability and cost—and demonstrates their application through a detailed image‑hosting service example, covering services, redundancy, partitioning, caching, proxies, indexing, load balancing, and queuing to achieve efficient, scalable data access.
CachingData PartitioningHigh Availability
0 likes · 37 min read
