This article reviews common primary key generation methods—including auto‑increment drawbacks and five alternatives such as UUID, step, segment, Snowflake, and Redis—explaining their mechanisms, advantages, and trade‑offs to help you select the best fit for high‑concurrency and distributed systems.
ElasticSearch achieves distributed search by organizing data into indices, types, mappings, documents, and fields, splitting indices into primary and replica shards across multiple nodes, with automatic master election and shard allocation, enabling horizontal scaling, high availability, and improved performance for large‑scale data workloads.
This article explains why MySQL tables with massive row counts suffer performance degradation and provides practical guidance on single‑table optimization, field and index design, query tuning, engine selection, system parameters, hardware upgrades, read‑write splitting, caching layers, partitioning, vertical and horizontal sharding, as well as client‑side and proxy‑side sharding solutions.
This article outlines core database architecture principles—high availability, performance, scalability, and consistency—examines four common deployment patterns (primary‑standby, dual‑primary, primary‑replica with read/write separation, and hybrid), and presents practical consistency solutions and personal insights for selecting and evolving database designs.
This article explains Redis Cluster’s automatic sharding, built‑in high availability, the role of the cluster bus ports, the limitations of traditional hash algorithms, and how consistent hashing with virtual nodes and the 16384‑slot hash slot mechanism enable efficient data distribution and low‑cost node scaling.
This guide walks through installing MySQL nodes, Java, and MyCat 1.6.5, configuring schemas, data nodes, sharding rules, adjusting ports, adding users, and verifying the MyCat service, providing complete code snippets and troubleshooting tips for a functional distributed database middleware setup.
This article explains Elasticsearch's architecture, including its reliance on Apache Lucene, shard and replica design, routing optimization, JVM garbage‑collection tuning, memory‑locking, and index‑merge control, offering practical guidance for building and operating high‑performance search clusters.
This article analyzes a community‑raised issue where a left join between a global table and a sharded table produces incorrect results in Mycat but works as expected in DBLE, explaining the underlying execution‑plan differences and how DBLE correctly merges results to avoid duplicate data.
This article explains Elasticsearch’s cluster and data layer architecture, detailing nodes, indices, shards, replicas, deployment models, and the trade‑offs of different distributed system designs, helping readers understand how Elasticsearch achieves scalability, reliability, and query performance.
This article explains MySQL replication (master‑slave and master‑master), data sharding principles and implementations, and various distributed database deployment architectures, covering replication mechanisms, advantages, failure recovery, sharding strategies, middleware like Mycat, and scaling solutions for high‑availability systems.
This article explains why deep pagination in Elasticsearch leads to high CPU usage, introduces core concepts such as indices, shards, nodes, and documents, and details the distributed CRUD operations, query and fetch phases, and segment management that affect search performance.
This article explains why a fast‑growing e‑commerce platform chose MongoDB, covering its high‑availability replica‑set architecture, Raft‑based election algorithm, replica‑set size limits, write‑concern trade‑offs, sharding components and load‑balancing, as well as the flexible document schema with practical code examples.
This article presents a thorough guide to improving MySQL performance for large single tables, covering field choices, indexing best practices, query tuning, engine selection, system parameters, hardware upgrades, read/write separation, caching layers, partitioning, vertical and horizontal sharding, and compatible scalable database alternatives.
This article explains what a search engine is, describes its core components—indexing and search modules—detailing the workflow from content acquisition to result rendering, and provides an in‑depth overview of Elasticsearch, including its architecture, clusters, shards, replicas, mappings, and basic configuration.
Redis is a high‑performance, open‑source key‑value store offering persistence, rich data structures, replication, pub/sub, transactions, and clustering; this guide explains its fundamentals, data types (string, hash, list, set, sorted set), essential commands, and advanced features such as persistence options, replication, Sentinel, and sharding.
This article explains MySQL performance bottlenecks, compares IO and CPU limits, introduces horizontal and vertical sharding and partitioning concepts, presents practical scenarios, outlines common tools such as Sharding‑Sphere, TDDL and Mycat, and provides step‑by‑step guidance for implementation, troubleshooting and scaling.
This article explains why Elasticsearch cross-index queries are essential, outlines their technical principles, showcases classic use cases such as business analytics, big‑data pipelines and log management, and provides practical methods, code examples, and performance considerations for effective implementation.
The article recaps the final DBLE open class session on May 12, presenting DBLE 3.0 roadmap, sharing PPT and video resources, and providing detailed Q&A covering features such as connection pooling, read‑write separation, distributed transactions, sharding, high‑availability, and future development plans.
Google Cloud engineer Jaana Dogan shares 17 hard‑earned database lessons—from ACID nuances and isolation levels to optimistic locking, sharding strategies, AUTOINCREMENT pitfalls, clock drift, and online migrations—providing a practical guide to help developers avoid common pitfalls and design more reliable data systems.
TDSQL enables Tencent Meeting’s non‑disruptive, linear horizontal scaling by using a stateless SQL engine, shard‑key selection, two‑phase distributed transactions, and one‑click set expansion, delivering near‑linear performance growth, high availability, and seamless capacity increase for millions of daily users worldwide.
This article explains how to design and scale a RabbitMQ cluster capable of handling millions of messages per second, covering core concepts, Google’s large‑scale test setup, sharding and federation plugins, mirror queues, reliability mechanisms, and practical tips for high‑availability and performance optimization.
This article explains database bottlenecks caused by IO and CPU limits, introduces horizontal and vertical sharding for databases and tables, compares popular sharding tools, discusses challenges such as distributed transactions, cross‑node joins, pagination and global ID generation, and offers guidance on when and how to apply sharding in real‑world systems.
This article objectively compares traditional sharding‑plus‑middleware architectures with modern NewSQL distributed databases, examining their design principles, distributed transaction handling, high‑availability mechanisms, scaling and sharding strategies, SQL support, storage engines, and overall maturity to help practitioners choose the most suitable solution for their workloads.
This article objectively compares NewSQL databases with middleware‑based sharding solutions, examining architecture, distributed transactions, CAP constraints, performance, high availability, scaling, SQL support, storage engines, maturity, and provides practical guidance for choosing the right approach.
When a MySQL table grows to millions of rows, query, insert, update and delete performance degrades sharply, but by applying proper field design, index strategies, SQL best practices, engine selection, system parameters, caching, partitioning, vertical and horizontal sharding, and choosing suitable client‑side or proxy‑side architectures, the database can efficiently handle tens of millions of records.
This article explains how to design and implement a cloud‑native, large‑scale distributed monitoring system using Prometheus, covering its limitations, service‑level sharding, centralized storage, federation, and high‑availability strategies to overcome scaling challenges in Kubernetes environments.
The article examines why upgrading a three‑shard MongoDB cluster from version 3.2 to 4.0 causes CPU usage to jump from 10 % to 40 %, attributing the increase to more frequent moveChunk operations and the associated deletion workload, and explains the underlying balancer threshold differences between the versions.
This article analyzes common database I/O and CPU bottlenecks, explains horizontal and vertical sharding concepts, presents practical sharding rules, discusses tooling such as Sharding‑Sphere, TDDL and Mycat, and outlines step‑by‑step procedures and common pitfalls for scaling relational databases.
This tutorial explains how to install MariaDB's Spider storage engine, create single-table remote links, configure users and servers, and leverage Spider for sharding and multi‑server table aggregation, providing practical code examples and performance considerations for distributed database setups.
Youzan’s Task Scheduling Platform (TSP) unifies its legacy cron and timeout systems into a modular, extensible framework that supports both timed and delayed tasks, offering per‑task rate limiting, sharding, isolation, retry policies, and a rich SDK, while its roadmap adds end‑to‑end monitoring, workflow orchestration, domain‑level clustering, transactional messaging, and dynamic task registration.
The article explains how to maintain transaction consistency between a primary database and sharded tables by using a flow‑log table and an asynchronous processor that ensures eventual consistency, handles ordering, retries, and avoids the complexity of transactional message queues.
The article explains how Citus enables sharding and MapReduce‑style parallel query execution in PostgreSQL, showing performance gains, example bucket algorithms, and how standard SQL can replace custom MapReduce code for large‑scale data analytics.
This article provides a comprehensive introduction to Redis, covering its key features, supported data structures, core commands, persistence mechanisms, replication, sentinel monitoring, sharding strategies, and transaction handling with practical code examples.
This article introduces Vitess, a cloud‑native MySQL sharding middleware, explains its key features, architecture, core concepts such as cells, keyspaces, shards and vindexes, and provides a step‑by‑step guide to performing a reshard from two to four shards while highlighting operational challenges and practical recommendations.
This article chronicles Dada's rapid growth from a single‑database backend to a multi‑layer architecture that employs MySQL read/write separation, vertical partitioning, and horizontal sharding with a custom ID generator to sustain millions of daily orders while maintaining performance and stability.
This article details how a high‑traffic MongoDB sharded cluster exceeding one million TPS was optimized through software‑level tweaks, configuration changes, WiredTiger storage‑engine tuning, and hardware upgrades, resulting in latency reductions from hundreds of milliseconds to a few milliseconds and significantly improved stability.
This article explains what Global tables are in DBLE, how to configure them in schema.xml, and demonstrates their behavior during various JOIN operations across sharded databases, highlighting benefits and performance considerations for low‑change, low‑concurrency dictionary data.
This article examines four common database architecture patterns—primary‑standby, dual‑primary, master‑slave read/write separation, and a hybrid dual‑primary + master‑slave—evaluating each against high availability, performance, consistency, and scalability, and then presents practical consistency‑resolution techniques and cache strategies.
This article surveys several MySQL middleware solutions—including Atlas, MySQL Router, Mycat, Cobar, Amoeba, and client‑side sharding—detailing their architectures, key features, advantages, and practical recommendations for production use.
This article explains how IO and CPU bottlenecks increase active database connections, describes horizontal and vertical sharding techniques, outlines practical steps and tools for implementing sharding, and discusses common challenges such as non‑partition key queries and expansion strategies.
The article explains why traditional sharding and database partitioning cannot guarantee limitless scalability, examines the connection‑limit problems of RPC‑based services, and introduces unitization as a practical approach to achieve true unlimited scaling while managing database connections efficiently.
After splitting a database into multiple shards, generating globally unique primary keys becomes essential, and this article examines various solutions—including auto‑increment IDs, sequence steps, UUIDs, timestamp concatenation, and the Snowflake algorithm—detailing their advantages, drawbacks, and suitable scenarios.
The article explains CB‑SQL's cloud‑native distributed database architecture, covering range‑based sharding, metadata management, hybrid logical clocks, pessimistic distributed transactions with SSI, Raft‑based strong consistency, and horizontal scalability for petabyte‑scale workloads.
This article objectively compares NewSQL distributed databases with middleware‑based sharding solutions, examining their architectures, distributed transaction handling, scalability, performance, high‑availability, and operational considerations, and provides guidance on selecting the appropriate approach based on workload, consistency, and organizational constraints.
This article provides a comprehensive overview of Elasticsearch, covering its role in handling structured and unstructured data, core concepts such as Lucene, inverted indexes, clusters, shards, replicas, mapping, indexing processes, storage mechanisms, and practical performance tuning tips for deployment.
After receiving a Young Generation GC alarm on a Pinduoduo service, I traced the issue to a high‑frequency scheduled task that created massive objects, then resolved it by breaking the job into finer‑grained tasks and finally sharding the work across multiple machines.
The article recounts a production incident where a GC alert triggered due to excessive object creation in a scheduled ad‑transaction sync task, and explains how the problem was diagnosed, mitigated by task splitting, and finally resolved through data sharding across multiple machines.
This article explains the architecture of Elasticsearch clusters, covering core concepts such as nodes, indices, shards, replicas, indexing flow, mixed and tiered deployment models, data‑layer storage, and compares these designs with other distributed data system architectures.
A MongoDB client reported severe chunk imbalance and nightly balancer migration failures in a sharded cluster, which were traced to a known bug causing conflicting operations, and the issue was resolved by disabling the balancer for the affected collection and upgrading the cluster to version 3.4.11 or later.
This article outlines core database architecture principles—high availability, performance, consistency, and scalability—then compares four common setups (primary‑backup, dual‑master, primary‑replica, and hybrid), analyzes their trade‑offs, and presents practical consistency and cache‑sync solutions along with personal insights.
Gaea is an open‑source MySQL middleware created by Xiaomi that replaces MyCAT, offering read‑write splitting, sharding, namespace‑based multi‑tenant configuration, Go‑driven architecture, hot‑reload via etcd, comprehensive monitoring, and a visual management console, with its source hosted on GitHub.
This article explains DBLE's hash‑based sharding algorithm that first computes a modulo to obtain a logical shard number and then directly maps it to a physical shard via a mapping table derived from partitionLength and partitionCount arrays, including configuration steps, development tips, operational guidance, and a comparison with MyCat.
QTSDB is a distributed time‑series database built on InfluxDB that adds cluster capabilities such as horizontal scaling, replica fault‑tolerance, sharding, meta‑service coordination, and comprehensive read/write processing, while also detailing failure recovery and node expansion strategies.
This article explains common I/O and CPU bottlenecks in databases, compares horizontal and vertical sharding techniques, introduces practical tools, outlines migration steps, and discusses typical challenges such as non‑partition‑key queries, cross‑shard pagination, and scaling.
This article presents a hybrid sharding approach that combines range partitioning to assign ID ranges to groups and hash modulo on the total number of tables to achieve uniform data distribution while avoiding hotspots and eliminating the need for data migration during scaling.
The article explains vertical and horizontal database sharding, compares hash-modulo and range-based partitioning schemes, discusses their advantages and drawbacks such as hotspot issues and migration challenges, and highlights the trade‑offs when scaling large‑scale order data.
The article explains common database performance bottlenecks such as I/O and CPU limits, and details horizontal and vertical sharding techniques—including database and table partitioning—along with tools, implementation steps, common issues, scaling strategies, and practical examples for improving scalability and reliability.
This article examines various distributed ID generation methods—including UUID, MySQL auto‑increment, Snowflake, and Redis—explaining their advantages, drawbacks, and how top companies combine block allocation and double‑buffering to achieve uniqueness, scalability, and high performance.
This article explains vertical and horizontal database partitioning, compares hash‑modulo and range sharding schemes, and proposes a hybrid approach that balances load, prevents hotspots, and eliminates the need for costly data migrations during scaling.
This article outlines core database architecture principles—high availability, performance, consistency, and scalability—examines four typical deployment schemes with their trade‑offs, and presents multiple consistency‑preserving strategies for both primary/replica and DB‑cache interactions.
This article shares a practical journey of handling MySQL tables that grew beyond a billion rows, describing a temporary backup‑table fix, a custom hash‑based sharding implementation, data migration tactics, validation steps, and key lessons learned for large‑scale database management.
This article explores practical techniques for handling massive, high‑concurrency data workloads, covering relational database limits, read/write separation, vertical and horizontal sharding, key selection, archival to NoSQL stores, and the use of heterogeneous index tables to maintain performance.
This guide explains how to improve MySQL database performance by identifying bottlenecks and applying both soft (query, indexing, table design) and hard (hardware, configuration, sharding, caching) optimizations, offering concrete commands, parameter settings, and practical examples.
Xianyu migrated its tens‑billion‑row product database from a shared MySQL cluster to an isolated instance by refactoring with dual‑write, employing TDDL for sharding and sequence management, and using Alibaba Cloud DTS for full‑load, incremental sync, consistency verification, and a no‑data‑loss rollback, achieving seamless, user‑transparent migration and improved stability.
This article introduces the concept of data heterogeneity in large-scale systems, explains how sharding creates query challenges, describes query‑dimension and aggregation heterogeneity, and details implementation techniques such as subscribing to MySQL binlog and using Alibaba’s Canal for reliable data synchronization and storage.
The announcement showcases a 30‑day community sharing event for the open‑source DBLE middleware, highlighting three member submissions that cover a cross‑shard query case study, detailed rule.xml and server.xml configuration guides, and a tutorial on creating custom sharding algorithms, along with reward details.
This article traces the evolution of Alibaba’s distributed data layer—from the early Cobar system to the modern TDDL framework and DRDS service—explaining their architectures, limitations, sharding principles, transaction‑boundary strategies, heterogeneous index tables, and the Jingwei data‑replication platform that together enable seamless scaling and high‑performance SQL processing across thousands of databases.
The article traces the evolution of backend services from monolithic applications through RPC-based scaling and database sharding, explains why sharding alone cannot achieve unlimited scalability due to connection limits, and proposes a unitized architecture that partitions databases per service to overcome these constraints.
This article outlines MongoDB's key features, describes a real‑world outage caused by misconfigured connection limits, details the root‑cause analysis and temporary remediation, and presents a comprehensive set of configuration, sharding, and hardware optimizations that dramatically improved the system's reliability and throughput.
The article details Youzan's three‑stage evolution of its order management system—from database sharding to Elasticsearch‑based cross‑table search and finally HBase for fast detail assembly—while addressing data sync, real‑time consistency, and idempotency strategies.
The article outlines eBay's key scalability best practices—including functional partitioning, horizontal sharding, avoiding distributed transactions, aggressive asynchronous decoupling, moving work to async pipelines, pervasive virtualization, and intelligent caching—to achieve linear or better resource usage as load grows.
This article examines various approaches to storing massive data, including table partitioning, NoSQL/NewSQL, and database sharding, analyzes their advantages and drawbacks, and presents practical sharding designs for user, forum, and order databases, with considerations for caching, redundancy, and query routing.
This article explains why modern applications dealing with massive user, order, and transaction data must move beyond single‑table designs, compares partitioning, sharding (分库分表), and NoSQL/NewSQL solutions, and provides practical guidance on choosing middleware, sharding columns, and hybrid architectures such as es + HBase.
Li Xiaohui shares Tencent Cloud MongoDB’s real‑world mini‑game operations, detailing schema‑free design, sharding, thread‑per‑connection tuning, snapshot‑based read fixes, and table‑level rollback, then demonstrates a one‑click cloud stack that provisions MongoDB, serverless functions, storage, monitoring and security for mini‑program developers.
This article chronicles the eight‑month development of ShardingSphere 3.0.0, detailing its origins from Sharding‑JDBC, the addition of Sharding‑Proxy and Sidecar, core feature enhancements, distributed transaction redesign, community growth, and future roadmap for the open‑source database middleware ecosystem.
This article introduces the free classic book “Cloud Design Patterns,” which catalogs 32 design patterns across eight problem domains—availability, data management, design & implementation, messaging, management & monitoring, performance & scalability, resiliency, and security—offering practical solutions and C# examples for modern cloud and distributed systems.
This article explains how to design a high‑throughput payment system using database sharding, Snowflake‑style globally unique order IDs, eventual consistency via message queues, high‑availability architectures, data tiering, and coarse‑fine traffic control to handle massive request spikes.
The article explains how distributed storage systems solve the fundamental problems of data sharding and redundancy by describing three sharding methods (hash, consistent‑hash, and range‑based), the criteria for choosing a shard key, the role of metadata servers, and consistency techniques such as leasing, all illustrated with concrete examples and code snippets.
The article examines MyCat’s middleware‑based pseudo‑distributed MySQL architecture, outlines its storage, scalability, and reliability shortcomings, walks through common solutions like disk expansion, compression, and sharding, and finally offers practical steps and alternative technologies for building truly distributed database systems.
This guide explains why MySQL performance drops when a single table grows huge and provides a comprehensive set of optimization steps—including field type choices, index design, query tuning, engine selection, system parameters, hardware scaling, read/write splitting, caching, partitioning, vertical and horizontal sharding, and architecture recommendations—to keep MySQL fast and reliable at scale.
Redis Cluster is a distributed system that partitions its keyspace into 16,384 hash slots across up to 16,384 nodes, enabling automatic sharding, slot migration, replication, and high‑availability features such as automatic node discovery, master‑slave election, and online resharding without service interruption.
This article describes how JD built a high‑concurrency price‑protection service for the 618 promotion, covering rate‑limiting, degradation, CDN and data caching, database sharding with smooth expansion, and a multi‑stage, fault‑tolerant task‑processing workflow to achieve unlimited scalability.
Elasticsearch is an open‑source, scalable, high‑availability distributed full‑text search engine that operates in near real‑time, using clusters of nodes, indexes, documents, shards and replicas to efficiently store and retrieve large volumes of data.
This article explains how to design data architecture for microservice systems, covering microservice fundamentals, advantages, decoupling, lightweight APIs, continuous delivery, database per service versus shared databases, polyglot persistence, scaling dimensions, sharding strategies, and why MongoDB is a suitable choice.
This article describes the background, analysis, solution selection, architecture redesign, execution steps, and results of a large‑scale order table splitting project that separates hot and cold data, applies horizontal sharding, and uses MySQL partitioning to control table size and boost query performance.
This article examines the requirements of active‑active applications across multiple data centers, compares distributed database architectures—including two‑phase commit, multi‑master, and sharded designs—and explains how MongoDB’s sharding and write/read concerns enable high availability, consistency, and rapid failover in such environments.
Telegram’s open‑source TON blockchain combines a master chain with dynamically sharded work chains, uses hyper‑cube routing for fast or reliable inter‑shard messages, employs PoS‑BFT consensus with validators, nominators and fishermen, supports services like DNS, Proxy, Storage and Pay, and manages a 5‑billion Gram token economy to enable scalable decentralized applications.
This article explains why and how relational databases should be sharded, covering vertical and horizontal partitioning techniques, common pitfalls such as distributed transactions and join queries, and practical guidelines for deciding when to split data and which tools to use.
This talk walks through the fundamentals of consumer‑finance services, details the evolving database architecture—including sharding, connection‑pool handling, resource isolation, and high‑performance caching—and shares the team's DevOps transformation and outlook toward NewSQL and AI‑driven operations.
The article examines how the rapid growth of mobile and aggregated payments strains traditional IOE‑based payment systems, and outlines a comprehensive distributed transformation strategy that includes vertical service‑oriented splitting, horizontal sharding by data type, and practical approaches to handling distributed transactions, cross‑shard queries, scaling, disaster recovery, and operational monitoring.
This article explains why distributed caching is essential for high‑traffic services, presents three programming approaches (direct code, Spring‑Data‑Redis injection, and annotation‑based), compares three cache‑access patterns, details sharding models, outlines smooth and offline migration steps, and addresses cache penetration, concurrency, avalanche, and transaction support with concrete code examples.
This article explains MyCat's role as a database middleware, tracing the evolution from single‑database setups through distributed deployment, caching, master‑slave, vertical and horizontal sharding, and shows how MyCat automates data‑source and shard selection to reduce code complexity.
This article reviews RadonDB, a MySQL‑based distributed database, detailing its architecture, SQL, storage, and compute nodes, sharding strategy, data‑balancing mechanisms, migration process, and practical recommendations for DBAs seeking high‑availability and consistent relational data handling.
This article examines the core technologies behind NewSQL databases for OLTP workloads, contrasting them with NoSQL and traditional relational systems by exploring storage engines such as B‑Tree and LSM‑Tree, sharding strategies, replication models, and distributed transaction mechanisms like two‑phase commit.
This article explores MySQL sharding fundamentals, detailing vertical, horizontal, and combined partitioning methods, their advantages and drawbacks, and surveys various open‑source implementations such as MySQL Proxy, HSCALE, Amoeba, HiveDB, and PL/Proxy, while offering practical guidance on configuration, transaction handling, and common pitfalls.
This article explains Elasticsearch’s cluster architecture, detailing node roles, index, shard, and replica concepts, compares mixed and tiered deployment models, examines data storage mechanisms, and discusses the advantages and trade‑offs of different distributed system designs.
This article explains how Qunar's international ticket pricing engine was optimized by redesigning hash rules for sharding, enhancing local cache replication, employing multi‑stage result delivery, and applying common distributed‑system patterns to boost computational capacity and reduce response latency for massive flight‑search workloads.
Database sharding, or splitting databases into multiple instances and tables, is essential for scaling backend systems; this article explains its importance, historical context, core concepts, storage stretching techniques, index and consistency handling, and provides a practical design example for transaction systems.
This article explains the motivations behind database sharding, defines the concept, explores its historical background, discusses storage stretching, indexing and consistency challenges, and presents a practical sharding design example for transaction systems.
This article details the evolution of JD's Elastic Database, describing the challenges of scaling MySQL, the staged solutions including sharding, JProxy, and the final elastic architecture with services like Topology, JED‑Gate, and JED‑Tablet, and explains its query processing, dynamic resharding, and automatic failover mechanisms.
