This article explains why MySQL queries become slow, how proper indexing, MDL locks, sharding, read‑write separation, and large‑table strategies can improve performance, and also introduces Elasticsearch and HBase as complementary technologies for search and high‑write workloads.
The article explains how Ant Group’s Logic Data Center (LDC) architecture evolved from monolithic to unit‑based distributed design, using zones, sharding, and read‑write separation to sustain a record 583,000 transactions per second during the Double‑11 shopping festival.
This guide introduces MyBatis‑Mate’s core capabilities—including dictionary binding, field encryption, data masking, automatic DDL maintenance, dynamic sharding, and data‑scope permissions—while providing Maven dependencies and complete Java code examples for integrating these features into Spring Boot applications.
The article explains how Facebook’s Shard Manager tackles large‑scale system sharding by combining stateful and stateless service deployment, consistent hashing versus sharding, fault‑as‑normal principles, replication, automated failover, load‑balancing, and elastic scaling to achieve high availability and efficient resource use.
This article describes how Ctrip's senior backend engineer designed and implemented horizontal database sharding, a service‑level proxy, dual‑read/write mechanisms, and a staged migration process to overcome order‑database bottlenecks, improve scalability, and ensure high availability for the rapidly growing international train‑ticket business.
This article outlines 58.com’s database architecture strategies—including high availability via replication and dual‑master setups, read‑performance enhancements through indexing, read replicas, and caching, consistency solutions, scalability techniques, horizontal sharding patterns, and a review of Codd’s twelve relational rules as design guidelines.
This guide explains how to build a highly available, scalable architecture for supporting hundreds of millions of users, covering decoupling, redundancy, vertical and horizontal scaling, load balancing, database replication, sharding, caching, CDN, GeoDNS, and best practices for progressive system expansion.
This article provides a comprehensive introduction to Elasticsearch, covering its underlying Lucene-based inverted index, data types, shard routing, cluster roles, discovery mechanisms, refresh and translog handling, segment merging, and practical performance and JVM tuning tips for building scalable, near‑real‑time search systems.
Version 2.3.0 of Loki introduces enhanced features such as per‑tenant custom retention policies, time‑range log deletion via the Compactor API, Prometheus‑style recording rules, a new pattern parser for LogQL, ingestion sharding for faster queries, and advanced IP‑matching syntax, all aimed at improving storage efficiency, compliance, and observability.
This comprehensive guide covers everything from MongoDB's core concepts, data model, and architecture to practical commands for creating databases, collections, indexes, sharding, replication, aggregation, and troubleshooting, providing clear answers to over fifty frequently asked questions.
This article provides a comprehensive overview of MongoDB, covering its definition, document‑oriented storage, key features, differences from relational databases, sharding and replication mechanisms, common commands, data types, indexing, aggregation, and practical usage scenarios for modern applications.
This article explains why MySQL chooses B+ trees for its indexes, covers the left‑most prefix rule for composite indexes, offers practical index‑design tips, discusses MyBatis caching, master‑slave replication, and sharding strategies for large‑scale databases.
The release of MyBatis‑Mate 1.0.1 introduces enterprise‑grade features such as dictionary binding, field encryption, data desensitization, automatic DDL maintenance, and dynamic multi‑datasource sharding, accompanied by code examples, future roadmap, and Maven dependencies for seamless integration into Spring Boot projects.
The article explains how IO and CPU bottlenecks increase active database connections, then details horizontal and vertical sharding techniques, practical tools, step‑by‑step implementation, and common pitfalls to help engineers relieve pressure on database resources.
This tutorial explains how to use Elastic Job's sharding configuration in Spring Boot to accelerate scheduled task execution, improve resource utilization, and achieve high‑availability by distributing work across multiple job instances.
This article presents a comprehensive overview of 58.com’s database architecture, covering availability through replication and dual‑master setups, read‑performance enhancements via indexing, replicas and caching, consistency strategies, rapid horizontal scaling, sharding patterns, and practical SQL considerations alongside Codd’s 12 relational rules.
This article outlines a comprehensive, step‑by‑step approach for migrating massive tables—up to billions of rows—through sharding, dual‑write, data synchronization, validation, testing, and controlled switch‑over to ensure minimal impact on production services.
The article outlines a range of techniques—including caching, read‑write splitting, vertical partitioning, sharding, cold‑hot data separation, Elasticsearch, NoSQL and NewSQL—to store and retrieve billions of user‑generated records efficiently while maintaining high throughput and low latency.
This article explains why MySQL queries become slow, how index design and common pitfalls affect performance, introduces MDL locks and large‑table strategies, then compares Elasticsearch and HBase as complementary storage and search solutions, providing practical code examples and best‑practice recommendations.
The article explains how MySQL uses different memory areas such as Thread Memory, Sharing, and InnoDB Buffer Pool, describes the impact of large user tables on query performance, and shows that when a table size exceeds the InnoDB buffer pool capacity, sharding becomes necessary.
This article explains why MySQL queries become slow, explores index-related pitfalls and optimization techniques, and then compares ElasticSearch and HBase as complementary solutions for large‑scale data and search scenarios, offering practical tips and code examples.
This article analyzes the typical evolution from monolithic to RPC-based services, explains why sharding and database partitioning cannot alone achieve unlimited scaling due to connection limits, and introduces a unitization approach that assigns each application to a specific database shard to overcome those constraints.
This article explains why MySQL queries become slow, how indexes work and fail, the impact of MDL locks, large‑table challenges, sharding and read‑write splitting strategies, then introduces Elasticsearch’s search capabilities and HBase’s column‑family storage, offering practical guidance for each technology.
This article traces the evolution of backend services from monolithic applications through RPC and sharding, explains why sharding and database partitioning cannot alone achieve unlimited scaling due to connection limits, and proposes a unitization approach that isolates database connections per service to enable true horizontal expansion.
This article analyzes common causes of slow MySQL queries such as index misuse, MDL locks, and large‑table bottlenecks, then presents practical solutions like proper indexing, sharding, read/write splitting, and evaluates when to complement MySQL with ElasticSearch or HBase for better performance.
Over four years at Ele.me, the author chronicles the transformation of the order platform from a single‑machine Python monolith to a distributed, sharded, message‑driven architecture, detailing challenges in scaling, performance testing, fault injection, Redis usage, and service separation that shaped today’s robust backend.
This article explains how to optimize Java scheduled tasks for large‑scale data synchronization by introducing phased improvements such as blacklist mechanisms, multithreaded execution with CountDownLatch, and server‑level sharding, providing practical guidelines for performance and reliability.
This article walks through Redis's architectural journey, covering single‑node basics, data persistence strategies (RDB and AOF), master‑slave replication, automatic failover with Sentinel, consensus algorithms, and both client‑side and server‑side sharding to achieve high performance and high availability.
This article presents a detailed case study of moving a 2‑petabyte smart‑product data workload from Elasticsearch to MongoDB, covering business background, source cluster architecture, MongoDB design and resource planning, step‑by‑step performance optimizations, latency and cost comparisons, and guidance on when each database is appropriate.
The article explains how to evolve a simple single‑instance database into a highly available, horizontally sharded system, detailing offline scaling, online dual‑write migration, and a smooth four‑node expansion process with configuration changes, data migration, and cleanup steps.
This article explains how to handle multi-table cross-database transactions by aligning sharding strategies for related tables, using a case study of shop_order_detail and shop_order_stat, and provides detailed formulas and TDDL configuration code to ensure consistent routing within the same database shard.
This article provides a comprehensive analysis of Redis Cluster, covering node and slot assignment, command execution, resharding, redirection, fault‑tolerance, gossip communication, scaling strategies, configuration limits, and practical code examples for building and operating a high‑availability sharded Redis deployment.
This article explains why MySQL uses B+‑tree indexes, describes the left‑most prefix rule for composite indexes, offers practical index‑design and optimization tips, outlines MyBatis first‑ and second‑level caching, details master‑slave replication, and introduces common sharding strategies and their implementation considerations.
This article explains the signs that indicate a need for database sharding, outlines performance bottlenecks, presents optimization techniques such as SQL tuning, table redesign, read‑write separation, and then details vertical and horizontal splitting strategies, their implementation across single or multiple databases, and the added complexities like cross‑database joins, distributed transactions, and ID generation.
This article examines core MySQL architecture principles—high availability, performance, consistency, and scalability—then compares four common deployment patterns, analyzes their trade‑offs, and presents practical consistency solutions for both master‑slave replication and cache synchronization, ending with personal insights on evolution and sharding.
The article outlines eBay’s practical scalability principles—including functional partitioning, horizontal sharding, avoiding distributed transactions, aggressive asynchronous decoupling, moving work to async pipelines, virtualization, and intelligent caching—demonstrating how large‑scale web systems can be designed for linear growth and high availability.
This article presents a comprehensive case study of Ctrip's unified accounting platform, detailing its evolution from legacy systems, architectural redesign, component decomposition, logging, sharding, asynchronous processing, database strategies, exception handling, and daily settlement mechanisms to achieve agility, decoupling, and reusability.
This comprehensive guide explains MongoDB cluster components, common use cases, monitoring commands, basic operational tasks, data migration procedures, troubleshooting of production issues, and optimization recommendations to achieve high performance and scalability.
This article explains how Vivo designed a company‑wide comment middle‑platform, chose MySQL and MongoDB, deep‑dived into MongoDB cluster architecture, shard key strategies, and practical solutions for scaling, migration, and high availability in a high‑traffic environment.
This article presents practical best‑practice guidelines for configuring Elasticsearch in production, covering heap memory sizing, CPU considerations, shard and replica planning, hot‑warm node architecture, node role settings, common pitfalls, monitoring APIs, and troubleshooting tips.
This article provides a comprehensive technical guide to distributed transactions, covering ACID fundamentals, consistency models, sharding techniques, CAP and BASE theories, and detailed implementations of two‑phase, three‑phase, and TCC protocols, while highlighting their advantages, limitations, and practical considerations.
This article explains the fundamentals of transparent sharding middleware in NewSQL systems, covering data partitioning strategies, read/write separation, the complete SQL processing pipeline, protocol adaptation for MySQL/PostgreSQL, distributed transaction models such as XA, Saga and TCC, and essential database governance and online scaling techniques.
To support Gaode Taxi’s soaring order volume, the team expanded from four to eight ECS instances, re‑sharded 256 tables into 4,096, built a custom binlog‑to‑Kafka sync middleware for full‑load and incremental migration, implemented rigorous validation and repair processes, and employed a gray‑rollout with ABC verification, completing the migration without code changes or incidents.
This article details Vivo's journey in building a company‑wide comment platform, explaining why MongoDB was chosen over MySQL, how its sharded cluster is structured, the challenges of shard‑key selection, and the practical steps taken to scale and maintain the system.
This article details the end‑to‑end process of expanding a high‑traffic ride‑hailing order system on Alibaba Cloud, covering capacity planning, database sharding from 256 to 4096 tables, custom data‑sync middleware, validation, repair, and a gray‑release migration that achieved zero incidents.
The article describes Uber’s Gairos platform—a unified real‑time data ingestion, storage, and query system built on Kafka, Elasticsearch and RT‑Gairos—detailing its architecture, Uber use cases such as dynamic pricing, the scalability and reliability challenges faced, and the suite of optimization techniques (sharding, query routing, caching, index merging, template tuning, and data‑pruning) applied to achieve low latency and high throughput.
This article objectively compares NewSQL distributed databases with traditional middleware‑plus‑sharding solutions, examining architecture, distributed transactions, CAP constraints, high‑availability, scaling, storage engines, maturity, and practical adoption scenarios to help readers decide which approach best fits their needs.
This article objectively compares NewSQL distributed databases with traditional middleware‑plus‑sharding solutions, examining their architectural differences, distributed transaction handling, high‑availability mechanisms, scaling and sharding strategies, SQL support, storage engines, and maturity to help practitioners decide which approach best fits their workload and operational constraints.
This article explains InnoDB's record storage, primary and secondary index mechanics, composite index rules, and provides practical e‑commerce sharding and table‑design guidelines with real‑world case studies to help developers optimize MySQL performance and storage efficiency.
This article systematically refutes ten widespread misconceptions about MongoDB—covering scalability, versioning, data format, transactions, relationships, sharding, security, data loss, and suitability—while explaining its architecture, BSON storage, and best‑practice usage with illustrative code examples.
This article explains common I/O and CPU bottlenecks in databases, introduces horizontal and vertical sharding techniques, discusses tools, global primary‑key generation methods, Snowflake ID algorithm, migration challenges, and provides practical guidance on when and how to apply database partitioning to improve performance and scalability.
This article objectively compares NewSQL distributed databases with middleware‑based sharding solutions, examining their architectural differences, distributed transaction handling, high‑availability mechanisms, scaling and sharding strategies, SQL support, storage engines, and overall maturity to help readers decide which approach best fits their workload.
This tutorial walks through setting up a Redis Cluster on a single server with six nodes, covering node startup, handshake, slot assignment, master‑slave replication, command routing, failover handling, and practical scaling operations such as adding, rebalancing, and removing nodes.
This article describes how the PLUS membership partner platform evolved from a simple initial design to a mature, high‑performance backend architecture by applying command, template, and strategy patterns, sharding‑JDBC, cluster separation, and Spring AOP for monitoring and exception handling.
When a MySQL table grows large, performance degrades sharply, so this guide explains single‑table tuning, proper field choices, index strategies, query best practices, engine differences, system parameters, hardware upgrades, read‑write splitting, caching layers, table partitioning, vertical and horizontal sharding, and how to choose suitable sharding solutions.
This guide explains how to boost MySQL performance through soft optimizations like query analysis, index usage, and table restructuring, as well as hard optimizations involving hardware upgrades, configuration tuning, sharding, read‑write splitting, and cache clustering.
The article details WeChat Search’s practical distributed architecture—using a Chubby‑elected leader for shard‑to‑node mapping, hash‑based sharding with dynamic rebalancing, a Lambda‑style batch and near‑real‑time indexing pipeline, relaxed monotonic consistency, and group‑based searcher scaling—to illustrate trade‑offs and lessons for building scalable, reliable search services.
This article details how Vivo transformed its monolithic e‑commerce order service into a scalable, service‑oriented system by applying data archiving, sharding‑JDBC based sharding, MySQL‑to‑Elasticsearch sync, zero‑downtime migration, and distributed‑transaction techniques, while sharing practical pitfalls and performance results.
This article explains how to leverage Flinkx and Flink Stream API to create a unified data‑sync task that extracts data from sharded MySQL tables, splits the workload, and pushes it to an MQ cluster, while detailing the underlying InputFormat and Reader architecture.
This article explains the concepts, types, advantages, and drawbacks of vertical and horizontal database sharding, discusses the challenges such as distributed transactions, cross‑node joins, pagination, global primary‑key generation, migration, and provides practical guidelines and case studies for deciding when and how to apply sharding.
This article provides a comprehensive guide to improving MySQL performance by optimizing single‑table schemas, indexes, queries, engine settings, system parameters, and then discusses read/write separation, caching layers, table partitioning, vertical and horizontal sharding, compatible scalable databases and when to consider NoSQL alternatives.
This article presents a comprehensive, step‑by‑step case study of a large‑scale MySQL sharding project, covering business refactoring, storage architecture design, data migration, synchronization strategies, capacity planning, validation, rollout procedures, stability safeguards, and cross‑team collaboration.
This comprehensive guide explains why and when to split relational databases, details vertical and horizontal sharding techniques, discusses associated challenges such as distributed transactions, cross‑node joins, pagination, global primary keys, and offers practical solutions and middleware options.
This article details how Vivo's cloud service tackled explosive data growth by applying horizontal and vertical sharding, routing‑table based dynamic expansion, and MySQL InnoDB compression, providing step‑by‑step guidance, performance results, and practical recommendations for large‑scale database deployments.
The article explains how Ant Financial’s Logical Data Center (LDC) and unit‑based architecture, combined with sharding, CAP analysis, and OceanBase’s Paxos‑based consensus, enable the payment platform to sustain tens of millions of transactions per second during Double‑11 events while ensuring high availability and disaster recovery.
This article explains the principles of hash tables, how hash functions are used for data sharding in distributed databases such as DBLE, compares hash‑modulo and consistent‑hash approaches, and discusses their advantages, limitations, and practical configuration tips.
This article examines common database performance bottlenecks such as I/O and CPU limits, explains the concepts, results, and scenarios of horizontal and vertical sharding—including database and table partitioning—and outlines tools, steps, and challenges for implementing effective sharding solutions.
This article analyzes common database performance bottlenecks such as I/O and CPU limits, explains how horizontal and vertical sharding (including database and table partitioning) can alleviate these issues, introduces practical tools and step‑by‑step implementation guidance, and discusses related challenges like non‑partition key queries and scaling.
This article dissects ClickHouse's Distributed table write path, detailing how the Distributed engine creates tables, decides between synchronous and asynchronous insertion, manages local and remote shards, handles sharding keys, and ensures atomic data distribution across the cluster.
This article explains the fundamentals of database sharding, covering both vertical and horizontal partitioning concepts, routing algorithms, advantages and drawbacks, practical implementation steps using sharding-jdbc, and discusses related challenges such as distributed transactions, pagination, global IDs, and tool selection.
This article systematically explains the fundamental concepts and typical pitfalls of distributed systems—including CAP and BASE theories, message‑queue reliability issues, distributed cache challenges, sharding strategies, and transaction models—while offering practical mitigation techniques for each problem.
Database read/write separation, alongside sharding and master‑slave architectures, mitigates high‑concurrency bottlenecks by distributing reads to replicas and writes to a primary, offering scalability and performance benefits while introducing challenges such as replication lag, routing complexity, and resource utilization trade‑offs.
This article explores the fundamentals of distributed systems, explaining what they are, why they’re used, design considerations such as replication and partitioning, the implications of the CAP theorem, common distribution strategies, typical architectural patterns, and the advantages and challenges of building and operating such systems.
This article provides a comprehensive, top‑down and bottom‑up explanation of Elasticsearch’s internal architecture, covering clusters, nodes, shards, Lucene segments, inverted indexes, stored fields, document values, caching, merging, routing, scaling, and query processing, while addressing common performance questions.
This article explains the internal workings of Elasticsearch, covering its cloud‑based cluster architecture, Lucene‑based indexing structures such as segments, shards, inverted indexes, stored fields and doc values, as well as search processing, caching, merging, routing, and scaling strategies.
The article examines why traditional sharding and database partitioning cannot alone achieve limitless scaling, explains the problem of excessive RPC‑to‑DB connections, and proposes a unitization approach that limits each service to a single database shard to enable true horizontal expansion.
This article explains Redis sharding, covering its purpose, various partitioning methods such as range and hash partitioning, implementation approaches like client‑side and proxy‑assisted sharding, their drawbacks, and practical steps for migration and using Redis Cluster or Twemproxy.
The article explains the fundamentals of distributed architecture, its high‑availability mechanisms, scalability benefits, common pitfalls such as data consistency, and real‑world case studies ranging from e‑commerce platforms to Nginx reverse proxy and microservice designs.
Sharding splits a single database into multiple servers to improve performance, covering vertical and horizontal partitioning, middleware options, and key challenges such as distributed transactions, cross‑node joins, ID generation, pagination, routing transparency, and choosing between frameworks or custom solutions.
This article reviews common primary key generation methods—including auto‑increment drawbacks and five alternatives such as UUID, step, segment, Snowflake, and Redis—explaining their mechanisms, advantages, and trade‑offs to help you select the best fit for high‑concurrency and distributed systems.
ElasticSearch achieves distributed search by organizing data into indices, types, mappings, documents, and fields, splitting indices into primary and replica shards across multiple nodes, with automatic master election and shard allocation, enabling horizontal scaling, high availability, and improved performance for large‑scale data workloads.
This article explains why MySQL tables with massive row counts suffer performance degradation and provides practical guidance on single‑table optimization, field and index design, query tuning, engine selection, system parameters, hardware upgrades, read‑write splitting, caching layers, partitioning, vertical and horizontal sharding, as well as client‑side and proxy‑side sharding solutions.
This article outlines core database architecture principles—high availability, performance, scalability, and consistency—examines four common deployment patterns (primary‑standby, dual‑primary, primary‑replica with read/write separation, and hybrid), and presents practical consistency solutions and personal insights for selecting and evolving database designs.
This article explains Redis Cluster’s automatic sharding, built‑in high availability, the role of the cluster bus ports, the limitations of traditional hash algorithms, and how consistent hashing with virtual nodes and the 16384‑slot hash slot mechanism enable efficient data distribution and low‑cost node scaling.
This guide walks through installing MySQL nodes, Java, and MyCat 1.6.5, configuring schemas, data nodes, sharding rules, adjusting ports, adding users, and verifying the MyCat service, providing complete code snippets and troubleshooting tips for a functional distributed database middleware setup.
This article explains Elasticsearch's architecture, including its reliance on Apache Lucene, shard and replica design, routing optimization, JVM garbage‑collection tuning, memory‑locking, and index‑merge control, offering practical guidance for building and operating high‑performance search clusters.
This article analyzes a community‑raised issue where a left join between a global table and a sharded table produces incorrect results in Mycat but works as expected in DBLE, explaining the underlying execution‑plan differences and how DBLE correctly merges results to avoid duplicate data.
This article explains Elasticsearch’s cluster and data layer architecture, detailing nodes, indices, shards, replicas, deployment models, and the trade‑offs of different distributed system designs, helping readers understand how Elasticsearch achieves scalability, reliability, and query performance.
This article explains MySQL replication (master‑slave and master‑master), data sharding principles and implementations, and various distributed database deployment architectures, covering replication mechanisms, advantages, failure recovery, sharding strategies, middleware like Mycat, and scaling solutions for high‑availability systems.
This article explains why deep pagination in Elasticsearch leads to high CPU usage, introduces core concepts such as indices, shards, nodes, and documents, and details the distributed CRUD operations, query and fetch phases, and segment management that affect search performance.
This article explains why a fast‑growing e‑commerce platform chose MongoDB, covering its high‑availability replica‑set architecture, Raft‑based election algorithm, replica‑set size limits, write‑concern trade‑offs, sharding components and load‑balancing, as well as the flexible document schema with practical code examples.
This article presents a thorough guide to improving MySQL performance for large single tables, covering field choices, indexing best practices, query tuning, engine selection, system parameters, hardware upgrades, read/write separation, caching layers, partitioning, vertical and horizontal sharding, and compatible scalable database alternatives.
This article explains what a search engine is, describes its core components—indexing and search modules—detailing the workflow from content acquisition to result rendering, and provides an in‑depth overview of Elasticsearch, including its architecture, clusters, shards, replicas, mappings, and basic configuration.
Redis is a high‑performance, open‑source key‑value store offering persistence, rich data structures, replication, pub/sub, transactions, and clustering; this guide explains its fundamentals, data types (string, hash, list, set, sorted set), essential commands, and advanced features such as persistence options, replication, Sentinel, and sharding.
This article explains MySQL performance bottlenecks, compares IO and CPU limits, introduces horizontal and vertical sharding and partitioning concepts, presents practical scenarios, outlines common tools such as Sharding‑Sphere, TDDL and Mycat, and provides step‑by‑step guidance for implementation, troubleshooting and scaling.
This article explains why Elasticsearch cross-index queries are essential, outlines their technical principles, showcases classic use cases such as business analytics, big‑data pipelines and log management, and provides practical methods, code examples, and performance considerations for effective implementation.
The article recaps the final DBLE open class session on May 12, presenting DBLE 3.0 roadmap, sharing PPT and video resources, and providing detailed Q&A covering features such as connection pooling, read‑write separation, distributed transactions, sharding, high‑availability, and future development plans.
Google Cloud engineer Jaana Dogan shares 17 hard‑earned database lessons—from ACID nuances and isolation levels to optimistic locking, sharding strategies, AUTOINCREMENT pitfalls, clock drift, and online migrations—providing a practical guide to help developers avoid common pitfalls and design more reliable data systems.
TDSQL enables Tencent Meeting’s non‑disruptive, linear horizontal scaling by using a stateless SQL engine, shard‑key selection, two‑phase distributed transactions, and one‑click set expansion, delivering near‑linear performance growth, high availability, and seamless capacity increase for millions of daily users worldwide.
This article explains how to design and scale a RabbitMQ cluster capable of handling millions of messages per second, covering core concepts, Google’s large‑scale test setup, sharding and federation plugins, mirror queues, reliability mechanisms, and practical tips for high‑availability and performance optimization.
