This comprehensive guide walks backend developers through practical techniques—such as proper indexing, SQL refactoring, parallel remote calls, asynchronous processing, caching layers, sharding, and monitoring—to diagnose and dramatically improve API response times and overall system throughput.
This article presents nine comprehensive performance‑optimization strategies—including load balancing, sharding, read/write separation, caching, indexing, CDN, asynchronous processing, code refinement, and algorithm improvement—aimed at boosting the efficiency and scalability of large‑scale backend systems.
This article details how Bilibili's Membership Store tackled massive traffic spikes by optimizing call chains, introducing asynchronous order processing, and implementing a sharding strategy that split databases and tables, ultimately achieving over 4000 TPS and stable performance during large‑scale promotions.
This article explains how Vitess was created to overcome MySQL leader‑follower replication limits at YouTube, detailing its sidecar VTTablet, stateless VTGate router, topology key‑value store, and scaling strategies that enable billions of users to be served reliably.
This article objectively compares NewSQL databases with middleware‑based sharding solutions, examining their architectures, distributed transaction handling, scalability, high‑availability mechanisms, storage engines, and suitability for various workloads, and provides guidance on selecting the appropriate approach based on specific system requirements.
This article shares a practical case study on how a coupon‑distribution system evolved from a single‑table design to a sharded, multi‑database architecture with query optimization, ElasticSearch integration, and read/write splitting to handle growing data volume, complex queries, and increasing request traffic.
Consistent hashing replaces simple modulo‑based distribution to efficiently locate cached data across changing numbers of servers, using a hash ring and virtual nodes to ensure balanced load, minimize data movement, and improve reliability in distributed caching, load balancing, and database sharding scenarios.
This article examines the open‑source XXL-JOB distributed task scheduler, explaining its lightweight architecture, time‑wheel based scheduling, consistent‑hash routing, sharding mechanisms, and includes annotated source code snippets to illustrate key implementation details for backend developers.
This article summarizes a live session where Shawn explains how understanding business scenarios drives effective OpenGemini database design, and provides comprehensive best‑practice guidance on library and table design, data ingestion, query optimization, and performance tuning for time‑series workloads.
This article explains database sharding concepts, reasons for using it, various sharding strategies, and demonstrates implementation with ShardingSphere, including configuration and Java code examples, while also offering additional resources for advanced architecture and interview preparation.
This article examines Pinterest's evolution through four development phases, the core technologies it adopted, and how it transitioned from complex clustering to a simpler sharding architecture to achieve scalable, high‑availability data storage.
This article outlines a comprehensive backend architecture for community platforms and billion‑level e‑commerce systems, covering microservice decomposition, DDD modeling, caching strategies, Redis clustering, message‑queue decoupling, database sharding, read‑write separation, distributed transaction approaches, multithreaded data migration, and massive counting techniques.
This article provides a comprehensive step‑by‑step guide to setting up MongoDB replica sets and sharded clusters, explaining the architecture, member roles, configuration files, initialization commands, and operational procedures for ensuring data redundancy, high availability, and horizontal scaling.
This article objectively compares NewSQL distributed databases with traditional middleware‑based sharding solutions, examining their architectures, distributed transaction support, high availability, scaling, storage engines, SQL capabilities, maturity, and suitability for various workloads, to help architects decide which approach best fits their needs.
This article objectively compares NewSQL databases with middleware‑based sharding solutions, examining their architectural differences, transaction handling, scalability, high‑availability mechanisms, storage engines, and maturity to help readers decide which approach best fits their specific workload and business requirements.
This article objectively compares NewSQL databases and middleware‑plus‑sharding architectures, examining their core principles, distributed transaction handling, high‑availability mechanisms, scaling and sharding strategies, SQL support, storage engines, and maturity to help engineers decide which solution fits their workload.
This article walks through Redis's architectural journey—from a lone instance to a high‑availability, high‑performance cluster—covering persistence (RDB, AOF, hybrid), master‑slave replication, Sentinel automatic failover, sharding strategies, and the modern Redis Cluster design.
This article walks you through Redis’s architectural journey—from a simple single‑node setup, through persistence mechanisms, master‑slave replication, Sentinel‑driven automatic failover, and finally sharding with Redis Cluster—explaining each component’s purpose, trade‑offs, and how they collectively boost performance and reliability.
This comprehensive tutorial walks you through MongoDB replica set concepts, member roles, primary election, configuration commands, as well as sharding fundamentals, shard key selection, cluster deployment, balancer management, and practical code examples for building a production‑grade database cluster.
This article objectively compares middleware‑based sharding with modern NewSQL distributed databases, examining their architectural differences, performance, transaction support, scalability, high‑availability, and operational considerations, to help practitioners decide which approach best fits their workload and organizational constraints.
This article explains Redis Cluster's sharding architecture, demonstrates how to configure multiple Redis nodes on different ports, shows commands for creating and testing the cluster, and illustrates failover behavior, highlighting its scalability and high‑availability advantages over Sentinel mode for large‑scale data workloads.
The article objectively compares NewSQL distributed databases with traditional middleware‑based sharding solutions, examining their architectural differences, distributed transaction support, performance, scalability, high‑availability mechanisms, storage engines, and practical suitability for various application scenarios.
This article provides a comprehensive, top‑down and bottom‑up explanation of ElasticSearch’s core architecture, detailing nodes, shards, Lucene segments, inverted indexes, stored fields, document values, caching, query processing, routing, and scaling considerations for efficient search operations.
Facing a 50‑million‑row financial transaction table that grew 6 million rows per month, the team analyzed the problem, set clear split goals, evaluated sharding middleware, designed a custom pagination algorithm, built a hybrid data‑migration plan, and executed a three‑stage rollout to safely replace the monolithic table with multiple 10‑million‑row shards.
This article presents a detailed case study of optimizing a distributed batch processing system by replacing static shard‑key concurrency with a dynamic task‑grabbing mechanism, dramatically reducing execution time from several hours to under fifteen minutes while maintaining stable resource usage.
The article presents a comprehensive case study of Ctrip's flight order itinerary system, detailing its background challenges, design goals, modular architecture, data aggregation strategies, sharding and Redis caching optimizations, and the adoption of Java structured concurrency to achieve high performance, reliability, and scalability.
This article introduces sharding-mybatis, a lightweight, non‑intrusive sharding library that seamlessly integrates with Spring Boot and MyBatis, explains why it may be preferable to Sharding‑Sphere, lists its features, provides a GitHub link, and offers detailed usage instructions with code examples.
This article examines how database middleware tackles the demanding needs of large‑scale internet services by providing centralized connection‑pool management, transparent read‑write splitting, diverse load‑balancing algorithms, sharding support, automatic failover, security controls, comprehensive monitoring, and flexible backup‑recovery mechanisms.
The article examines Oracle’s evolution from RAC‑based sharding to the 23c Global Distributed Database, highlighting its RAFT replication, cross‑region capabilities, compliance benefits, and performance advantages that position it as a true multi‑cloud, multi‑location solution for modern OLTP workloads.
This article presents a real‑world case study of optimizing massive batch‑processing tasks in an ad‑platform by applying query‑level improvements, cursor‑based pagination, shard‑aware batch updates, JVM‑tuned garbage collection, and distributed load‑balancing, ultimately reducing CPU usage from 80% to under 2% and cutting query‑per‑minute volume from millions to a few thousand.
This article explains how to build an efficient in‑memory cache in Go by starting with a basic map, adding read‑write locks for concurrency, reducing lock contention through sharding, minimizing GC overhead with a ring buffer, and finally using the high‑performance BigCache library.
This article explains how to select appropriate sharding keys based on business requirements and data access patterns, describes vertical and horizontal partitioning methods, and presents common techniques such as equal‑value, index, and gene methods along with key generation strategies like auto‑increment, UUID, Snowflake algorithm, and open‑source middleware.
This article details how Bilibili’s membership‑purchase platform tackled massive traffic spikes by redesigning the order call chain, introducing concurrent and asynchronous processing, and applying a sharding strategy that split databases and tables, ultimately boosting latency performance and supporting over 4,000 TPS during peak sales.
This guide explains how to design a high‑concurrency, high‑throughput internet architecture that ensures database high availability with double‑master sync and virtual IPs, and how to horizontally shard and smoothly expand the cluster in seconds using configuration changes, reloads, and cleanup steps.
The article explains why traditional MySQL sharding is problematic, introduces the open‑source distributed database TiDB, details its architecture, core features, use cases, performance benchmarks against MySQL, and provides guidance on when to adopt TiDB for large‑scale workloads.
This article details the end‑to‑end design of a payment system, covering transaction flow, horizontal and vertical pre‑optimizations, task scheduling, sharding strategies, data structures, high‑availability mechanisms such as channel isolation and Hystrix, and future planning for dynamic scaling and intelligent routing.
This article explains how to assess MySQL table size through table capacity, disk usage, and instance resources, illustrates the impact of B+‑tree index depth on query speed, and presents three practical solutions—table partitioning, database sharding, and hot‑cold data archiving—to improve performance.
This article introduces the concept of database sharding, explains how sharding middleware improves scalability and performance, and reviews six popular middleware options—ShardingSphere, MyCAT, TDDL/DRDS, Atlas, Vitess, and Oceanus—helping engineers choose the right tool for billion‑row datasets.
Bilibili’s membership purchase system was re‑engineered by refactoring serial service calls into a responsibility‑chain with parallelism, moving weak dependencies to asynchronous queues, and implementing sharding across 256 tables, boosting peak throughput from ~600 QPS to over 4,000 TPS and eliminating latency incidents during massive promotional traffic spikes.
This article outlines why databases become slow, introduces a four‑layer optimization framework, and presents eight practical solutions—including data reduction, archiving, sharding, caching, and read‑replica strategies—to help engineers systematically diagnose and resolve the majority of performance bottlenecks.
This article introduces the essential concepts of sharding architecture—including data partitioning, logical and physical tables, sharding keys, routing strategies, SQL rewriting, distributed primary keys, data masking, transactions, migration, and shadow databases—providing a comprehensive guide for building high‑performance, horizontally scaled database systems.
This article outlines eight comprehensive database optimization approaches—reducing data volume, using space for performance, selecting appropriate storage systems, and related techniques such as data archiving, intermediate tables, serialization, sharding, distributed caching, read‑write splitting, and CQRS—to help backend engineers systematically address common performance bottlenecks.
This article examines the current landscape of Chinese databases, compares centralized and distributed architectures, outlines when distributed solutions are truly needed, provides performance test data, and offers practical advice on sharding, SQL design, and avoiding cross‑node bottlenecks.
This tutorial walks through setting up a PostgreSQL 16.1 + Citus 12.1 cluster with Docker, creating separate schemas and roles for user, time, and ping microservices, distributing them via schema‑based sharding, and demonstrating CRUD operations and verification of data placement across three worker nodes.
This article explains the fundamentals of scheduled tasks, highlights the limitations of single‑node solutions, introduces the open‑source XXL‑JOB platform, walks through its architecture, installation, configuration, task creation, routing strategies, clustering, sharding and GLUE mode, and offers practical code examples for Java/Spring‑Boot projects.
This article walks through implementing EasyCache, a Go‑based in‑memory cache with sharding, lock‑free concurrency, LRU eviction, and configurable expiration, explaining the underlying data structures, goroutine cleanup logic, and key handling with concrete code examples.
This guide explains what a sharded PostgreSQL cluster is, how StackGres implements it with Citus and Patroni, and provides step‑by‑step YAML manifests and scripts to create both a basic and a custom production‑grade high‑availability sharded cluster on Kubernetes.
To import one billion 1 KB log records stored in HDFS or S3 into MySQL as quickly as possible, the article analyzes constraints, evaluates single‑table limits, proposes sharding, batch inserts, storage‑engine choices, file‑reading strategies, task coordination, reliability mechanisms, and concurrency control.
This article examines the trade‑offs between centralized and distributed OLTP databases, presents industry usage statistics, performance benchmarks, practical questions for migration, and detailed guidance on sharding, SQL design, and operational considerations to help decide when a distributed solution is truly needed.
This article explains why database sharding is needed, the problems caused by oversized databases and tables, and how to apply vertical and horizontal splitting using MyCat, including configuration examples, splitting strategies, and testing procedures.
This tutorial walks you through adding MongoDB dependencies to a Spring Boot project, configuring the connection, defining entity and repository classes, implementing service and controller layers, and optionally setting up sharding or replica‑set architectures for scalable data management.
This article explains the principles and practical designs of high‑performance database clusters, covering read/write separation, master‑slave replication lag, distribution mechanisms, sharding (both vertical and horizontal), business‑level partitioning, and the role of NoSQL technologies such as key‑value stores, document databases, columnar databases, and full‑text search engines.
This article presents five expansion strategies—shutdown, write‑stop, log‑based, dual‑write, and smooth 2N—detailing step‑by‑step procedures for MariaDB installation, master‑master replication, dynamic data‑source configuration, and Keepalived high‑availability setup, enabling seamless horizontal scaling and minimal service disruption for large‑scale databases.
The article explains that while MySQL foreign keys ensure data consistency and integrity, they introduce performance overhead, lock contention, and scalability issues—especially in high‑concurrency and sharded environments—leading many large internet firms to forego them in favor of application‑level solutions.
This article explains how to evaluate MySQL table size, understand B+‑tree limitations, and improve query performance by using table partitioning, horizontal/vertical sharding, and hot‑cold data archiving, while also comparing the trade‑offs of each approach for practical selection.
This article analyzes the constraints of loading one billion 1 KB log records from distributed storage into MySQL, evaluates single‑table limits, proposes batch‑insert, sharding, storage‑engine, file‑reading, and distributed‑task coordination techniques to achieve high‑speed, ordered, and reliable data ingestion.
When MySQL tables grow to millions of rows, insert and query latency increase, schema changes become costly, and irrelevant data bloats storage, so this guide evaluates table size, explains why B+‑tree depth hurts performance, and presents partitioning, sharding, and hot‑cold archiving solutions with practical SQL examples.
This article explains MongoDB's three cluster architectures—deprecated Master‑Slave replication, the highly available Replica Set with automatic failover, and horizontal scaling Sharding—detailing their roles, data flow, advantages, limitations, and operational considerations for building reliable, scalable databases.
The article outlines eBay’s key scalability best practices—including functional partitioning, horizontal sharding, eliminating distributed transactions, aggressive asynchronous decoupling, intelligent caching, and pervasive virtualization—to illustrate how large‑scale web systems can achieve linear or sub‑linear growth while maintaining availability and performance.
This article explains database sharding concepts, compares vertical and horizontal partitioning, details ID‑range and modulo sharding methods, discusses the read‑amplification issue caused by non‑shard queries, and presents solutions such as auxiliary index tables, Elasticsearch integration, and TiDB as alternatives.
This article walks through essential techniques for building high‑concurrency systems, covering page static‑generation, CDN acceleration, caching layers, asynchronous processing with thread pools and MQ, sharding, connection pooling, read/write splitting, indexing, batch queries, clustering, load‑balancing, rate limiting, service degradation, failover, multi‑active deployment, stress testing, and monitoring.
This article presents a hands‑on walkthrough of implementing in‑database horizontal partitioning by month for a high‑volume billing table, covering the business background, design decisions, stored‑procedure table creation, dynamic table selection in MyBatis, and query strategies such as serial‑number lookup and date‑range retrieval.
This article breaks down real‑world high‑concurrency strategies—horizontal scaling, caching, Elasticsearch, sharding, message‑queue smoothing, and cellization—explaining when each applies, their trade‑offs, and practical tips for building scalable, reliable backend services.
This article explains why relational databases hit performance bottlenecks at large scale, introduces vertical and horizontal sharding techniques, compares their pros and cons, discusses common challenges such as distributed transactions, joins, pagination and global key generation, and offers practical solutions and middleware options.
MongoDB is an open‑source, C++‑based NoSQL document database that stores data as BSON documents in collections, offering schema‑less modeling, embedded structures, high scalability through sharding, and a rich set of CRUD commands for modern application needs.
The article analyzes why massive tables become a bottleneck in Spring Boot systems, outlines the drawbacks of sharding, and presents a layered solution—data archiving, read‑write separation with caching, heterogeneous source synchronization via Elasticsearch and Canal, and selective sharding—to enable high‑performance real‑time search.
The article examines a 7‑node Elasticsearch cluster with 500 million documents, uncovering excessive heap usage, high OS memory pressure, numerous deleted documents, large translog, low query‑cache hit rate, and an over‑sharded design, then offers concrete tuning and redesign recommendations to restore performance.
This article defines distributed databases, examines their external traits such as write‑heavy, low‑latency, massive concurrency, massive storage and high reliability, explores internal architectures like client‑side sharding, proxy middleware and unit‑based designs, compares them with Amazon Aurora, and summarizes key takeaways.
This article explains how Qunar’s ticket ancillary service upgraded from a single-database architecture to a sharded one without downtime, detailing dual-write, transaction handling, mapping-key routing, diff verification, the challenges encountered, and a component-based solution that makes future migrations reusable.
The article explains that the core of the programmer's 35‑year crisis is the rapid pace of technological change outstripping learning speed, illustrated with backend sharding, frontend toolchain shifts, and AI advances, and offers practical advice to broaden skills beyond narrow tech stacks.
This article analyzes a 7‑node Elasticsearch cluster with 500 million documents, revealing excessive shard count, high heap and OS memory usage, large translog, low query‑cache hit rate, and heavy I/O, and offers concrete recommendations on mapping, sharding, JVM tuning, and resource management to restore performance.
This article walks through Redis's evolution from a simple single‑instance cache to a highly available, high‑performance cluster, explaining persistence mechanisms (RDB, AOF, hybrid), master‑slave replication, Sentinel automatic failover, and sharding strategies with concrete examples and trade‑offs.
This article explains the AKF Cube methodology for scaling distributed systems, detailing how horizontal replication (X axis), functional decomposition (Y axis), and user‑based sharding (Z axis) can be combined to boost performance while balancing implementation costs.
This tutorial walks through the fundamentals and architecture of ShardingSphere, compares it with other sharding tools, and demonstrates how to implement database sharding in a Spring Boot project using both YML configuration and pure Java code, including detailed code snippets and deployment tips.
This article explains how to smoothly migrate a legacy single‑database system to a sharded architecture by using dual‑write, mapping‑key routing, custom transaction handling, and offline/real‑time diff checks, while detailing the underlying MyBatis integration and component‑based design.
This article chronicles JD Baitiao’s journey from early MySQL and NoSQL solutions through DBRep to the adoption of Apache ShardingSphere, highlighting the technical motivations, decoupling strategies, performance comparisons, and the broader Database Plus vision for scalable, stable financial‑grade data architectures.
The DBLE 3.23.04.0 release introduces six new or refactored features, fixes over five critical community issues and more than a hundred bugs, adjusts TCP keep‑alive parameters, adds experimental configuration support, and provides detailed LTS version updates and incompatibility notes for users.
This article explores the challenges of distributed join queries, detailing MySQL sharding‑jdbc join implementation, routing strategies, and code examples, then examines Elasticsearch‑SQL join capabilities, various join algorithms, and practical considerations for using nested types, offering insights for optimizing performance in distributed data environments.
This article describes the challenges of migrating a long‑running monolithic database to a sharded architecture, presents a two‑stage smooth migration strategy using double‑write, diff verification, and mapping‑key techniques, and details the componentized design, core implementation, and configuration of a reusable sharding framework built on MyBatis and Spring.
Facing a 50‑million‑row financial transaction table, the team implemented horizontal sharding with sharding‑jdbc, designed migration and pagination strategies, tackled multi‑datasource transaction issues, and outlined a phased rollout, providing practical insights for large‑scale MySQL table splitting and backend system stability.
This article recounts how a finance team tackled a 50‑million‑row MySQL table by analyzing system bottlenecks, defining splitting goals, selecting sharding‑jdbc, addressing multi‑datasource transactions and pagination, designing data migration strategies, and executing a staged rollout to ensure stability and performance.
This article introduces the fundamental concepts of database sharding, including data fragmentation, nodes, logical and physical tables, broadcast and single tables, sharding keys, strategies, algorithms, binding tables, SQL parsing, routing, rewriting, execution, result merging, distributed primary keys, data masking, distributed transactions, migration, and shadow databases, providing a comprehensive guide for designing and operating a sharded architecture.
This article details how a finance team tackled a 50‑million‑row MySQL transaction table by designing a horizontal sharding strategy with Sharding‑JDBC, addressing performance bottlenecks, multi‑source transaction handling, pagination across shards, and a phased data migration and rollout plan.
The article details how a finance team tackled a 50‑million‑row MySQL transaction table by analyzing the pre‑split problems, defining split goals, choosing sharding‑jdbc, addressing multi‑datasource transaction and pagination issues, designing a phased migration and rollout plan, and summarizing lessons learned.
This article compares common database sharding strategies—range sharding, mapping tables, ID modulo, hash‑based sharding, and consistent hashing—explaining their implementation complexity, data distribution characteristics, scalability concerns, and practical trade‑offs with code examples.
This article explains common business scenarios such as large‑table splitting, cross‑database migration, and data synchronization, then compares loss‑bearing and loss‑less migration strategies, and provides detailed step‑by‑step implementation guidance for smooth dual‑write migration and real‑world tooling.
This comprehensive guide explains MongoDB cluster architecture, component roles, common use cases, monitoring commands, essential maintenance operations, data migration steps, troubleshooting of typical production issues, and practical optimization recommendations for high‑performance deployments.
This technical guide explains how to scale a sharded database horizontally by introducing five expansion schemes—including shutdown, write‑stop, log‑based, dual‑write, and smooth 2N approaches—covers MariaDB installation, master‑master replication setup, dynamic data‑source configuration with ShardingJDBC, and detailed Keepalived high‑availability configuration for seamless service continuity.
This article explains how to assess MySQL table size, identify performance issues caused by massive data volumes, and presents three practical solutions—table partitioning, database sharding, and hot‑cold data archiving—along with detailed SQL examples and considerations for implementation.
This article explains the concepts of database sharding, compares vertical and horizontal partitioning, discusses ID‑range and modulo sharding methods, analyzes the read‑amplification problem caused by non‑sharding keys, and presents practical solutions such as auxiliary index tables, Elasticsearch integration, and TiDB adoption.
This article explains why large MySQL tables cause slow queries, analyzes table size, disk usage, and instance capacity, and presents three practical solutions—table partitioning, database sharding (horizontal/vertical), and hot‑cold data archiving—helping developers choose the appropriate strategy for their workloads.
This article explains the concepts, strategies, and practical scenarios for MySQL database partitioning, covering both table splitting (vertical and horizontal) and database splitting (vertical and horizontal), with examples, advantages, and when to apply each technique to handle large-scale data workloads.
This article explains how to assess MySQL table size, why massive tables degrade performance due to B+‑tree depth, and presents three practical solutions—table partitioning, database sharding, and hot‑cold data archiving—along with guidance on selecting the appropriate approach.
This article explains how Apache Doris implements data sharding, partitioning (both list and range, static and dynamic), batch partition creation, and the Auto Bucket feature, providing detailed syntax, algorithms, and performance benefits for large‑scale distributed databases.
This comprehensive guide explores essential backend techniques—including lock‑free programming, zero‑copy I/O, efficient serialization, pooling, concurrency, async processing, caching strategies, sharding, storage optimizations, and queue mechanisms—to build high‑performance, scalable services while highlighting practical code examples and real‑world trade‑offs.
This guide walks developers through transforming a single‑node Redis cache into a stable, high‑performance cluster by explaining data persistence options (RDB, AOF, hybrid), master‑slave replication, Sentinel‑driven automatic failover, and both client‑side and proxy‑based sharding for horizontal scalability.
This article provides a comprehensive technical overview of Elasticsearch, covering core concepts such as indexes, types, documents, fields, mappings, shards and replicas, the cluster architecture, routing, write and read workflows, segment handling, translog, refresh and flush mechanisms, as well as practical optimization recommendations for hardware, shard strategy, routing, write performance and memory configuration.
The article explains the limitations of standalone MySQL for large‑scale workloads, explores sharding, read/write separation, and middleware approaches, and argues that modern distributed databases and cloud‑native solutions are the sustainable path forward for data‑intensive applications.
This comprehensive guide explains what Elasticsearch does, its underlying Lucene technology, core concepts such as clusters, shards, replicas, mapping, indexing and storage mechanisms, and provides practical performance‑tuning advice for building and operating a robust distributed search engine.
The article explains how single‑instance MySQL struggles with huge tables, resource‑starved databases, and high read traffic, and argues that sharding middleware is being replaced by modern distributed databases such as TiDB, OceanBase, Aurora, and PolarDB as the sustainable scaling solution.
