This guide explains MySQL partitioning fundamentals, shows how to create RANGE, LIST, and KEY partitions with concrete CREATE TABLE examples, compares the performance benefits and maintenance trade‑offs, and clarifies the differences between partitioning and sharding to help you decide when to use each technique.
This article explains why using LIMIT on sharded tables can crash databases, then introduces three practical pagination tricks—anchor pagination, shard‑mark method, and reverse pagination—detailing their workflows, performance benefits, and common pitfalls for backend developers.
The article explains why interviewers often reject flashy project tech stacks, shows how to evaluate the practicality of your own solutions, and lists common over‑engineered patterns such as unnecessary micro‑service suites, misuse of Redis, and needless sharding.
This article walks through the end‑to‑end system design of a TikTok‑style video leaderboard, covering functional and non‑functional requirements, traffic estimation, single‑node prototypes, fault‑tolerant replication, sharding, elastic partitioning, sliding‑window counters, and caching strategies to achieve low‑latency, high‑throughput ranking at massive scale.
The article outlines five Redis deployment patterns—from a single‑instance setup to official Redis Cluster—detailing their architectures, advantages, drawbacks, and suitable scenarios, helping engineers choose the right solution for scalability, high availability, and operational complexity.
This article explains Redis’s three clustering options—master‑slave replication, Sentinel high‑availability, and sharding—detailing their architectures, setup steps, synchronization mechanisms, advantages, drawbacks, and common interview questions, helping readers choose and implement the right solution for high‑performance, scalable data storage.
This article dissects the challenges of implementing pagination on billion‑row sharded tables, explains common sharding strategies, compares SDK, Proxy, and Sidecar architectures, and presents practical solutions such as global query, infinite scroll, two‑phase queries, index tables, and external storage to help engineers ace interview questions.
This article explains why horizontal sharding becomes essential for billion‑row tables, introduces the non‑intrusive GORM Sharding plugin, walks through installation, configuration, and code examples, and evaluates its advantages, limitations, common issues, and suitable use cases for Go back‑end services.
The article explains why the last six digits of Taobao order numbers are constant, revealing the “gene” method that embeds a hashed user identifier to bind orders to specific shards, thereby improving query performance, ensuring balanced data distribution, and supporting scalability in large e‑commerce systems.
The article explains how Taobao embeds a user‑specific “gene” in the last six digits of its order numbers, enabling efficient sharding routing, uniform data distribution, and idempotent order handling while maintaining global uniqueness in a large e‑commerce system.
The article details a real‑world case of splitting a massive MySQL financial flow table, covering the initial problems, sharding strategy selection, technical challenges like multi‑datasource transactions and pagination, data migration plans, and a phased rollout to ensure system stability.
This article explores ClickHouse’s high‑performance OLAP design—including its MPP architecture, columnar storage, vectorized execution, pre‑sorting, sharding, replication, index strategies, and compute engine—showing how each innovation contributes to ultra‑fast, scalable data analysis in the big‑data era.
This article explains Elasticsearch's core architecture including distributed sharding, near real-time search, inverted indexes, and key concepts like indices, documents, mappings, and replicas, along with typical use cases and the Elastic Stack ecosystem.
This article deeply analyzes ShardingSphere-JDBC 5.3.0 source code, explaining its JDBC‑compatible sharding design, modular architecture, adapter pattern implementation, SQL parsing and routing, distributed transaction support, elastic scaling, performance optimizations, and extensibility via SPI.
This guide introduces the ELK stack fundamentals, explains Elasticsearch, Logstash, and Kibana roles, walks through environment preparation, installation, configuration, head plugin setup, shard and replica concepts, scaling recommendations, and provides scripts for monitoring cluster health, offering a comprehensive hands‑on reference for log analytics operations.
This comprehensive guide explains MongoDB sharding fundamentals, walks through step‑by‑step deployment of config servers, shard replica sets, and mongos routers, compares range, hash, and compound shard keys, and provides performance tuning, security, backup, monitoring, and troubleshooting best practices for production‑grade clusters.
The article analyzes high‑availability concepts, from single‑machine risks to multi‑site active‑active architectures, compares cold and hot backup strategies, discusses network latency challenges, and presents Ele.me’s cell‑based, sharding‑driven multi‑region solution with concrete examples, tables, and code snippets.
This comprehensive guide walks MySQL administrators through diagnosing slow queries, configuring the slow‑query log, leveraging EXPLAIN, optimizing indexes, rewriting joins, implementing pagination tricks, and scaling with read‑write splitting, master‑slave replication, and horizontal sharding to dramatically improve performance in high‑traffic production environments.
This article walks through the end‑to‑end design, configuration, and implementation of splitting massive loan and repayment tables into 50 sharded MySQL tables using ShardingSphere, Spring Boot, and a custom suffix algorithm, while covering data migration, DBA coordination, dynamic switches, and scheduled consistency checks.
This article breaks down the massive‑scale challenge of calculating mutual friends for hundreds of millions of users, analyzes data‑size, latency, and memory constraints, and presents three Redis‑based solutions—basic Set, bitmap, and a sharded architecture—complete with code snippets, performance numbers, and practical trade‑offs.
The article walks through why sharding is needed, outlines IO and CPU bottlenecks, presents two design principles, shows how to estimate capacity from existing data and growth trends, compares range, modulo, consistent‑hash and Snowflake sharding schemes, and details migration strategies for expanding nodes without downtime.
This article provides a comprehensive overview of ElasticSearch, covering its role as a distributed full‑text search engine built on Lucene, key concepts such as index, type, document, field, shard and replica, the analysis pipeline, inverted index mechanics, and the two‑phase query‑fetch search workflow.
This article explores the performance bottlenecks of a 700‑million‑row MySQL orders table and presents a comprehensive sharding solution—including vertical and horizontal partitioning, gene‑based shard keys, routing logic, global secondary indexes, data migration strategies, and common pitfalls—demonstrating how query latency can drop from seconds to milliseconds while supporting billions of records.
This article explains how to handle MySQL order tables that have grown to billions of rows by applying vertical and horizontal sharding, gene‑based partitioning, cross‑shard query strategies, data migration techniques, and practical pitfalls to achieve massive performance improvements.
This article explores the pain points of a 700‑million‑row MySQL order table, presents vertical and horizontal sharding strategies, introduces gene‑based Snowflake IDs, details routing logic, migration steps, common pitfalls, and shows performance gains after applying the final architecture.
This article provides a comprehensive overview of ElasticSearch, covering its definition, core components such as indexes, shards and replicas, the analysis pipeline, inverted index mechanics, and the two‑stage search process that enables scalable, fault‑tolerant full‑text search in big‑data environments.
This article walks through the complete process of designing, configuring, and coding a sharding solution for loan and repayment tables—including database basics, historical data migration, backend query refactoring, a three‑write strategy, scheduled consistency checks, and full SpringBoot/ShardingSphere implementation details.
This article systematically analyzes a Spring Boot project's 280‑second startup, identifies bottlenecks such as bean initialization and sharding data source loading, and applies listener‑based timing, bean‑post‑processor profiling, and asynchronous initialization to reduce launch time to 159 seconds, improving developer efficiency.
This article examines the challenges of high‑concurrency write scenarios in ad‑billing systems, explains why row‑level locking becomes a bottleneck, and presents a step‑by‑step solution using Kafka for asynchronous peak shaving, parallel consumer processing, database sharding, hotspot dispersion, and batch deduction to achieve reliable real‑time balance updates.
This article explains how Baidu's vertical search platform tackled massive data growth and scaling challenges by redesigning its data management system, introducing elastic scheduling, decoupling ETCD access, implementing auto‑scaling, and advancing shard expansion to improve performance, stability, and cost efficiency.
This article explains how to locate and analyze MySQL slow queries, understand index structures and types, apply best practices for index creation, handle large pagination, manage transaction isolation and MVCC, and implement master‑slave replication and sharding strategies for high‑performance databases.
This article explains a metadata‑driven approach for extending massive tables—using a separate extension table, sharding, and Elasticsearch sync—to add new fields to billion‑row databases without locking the primary table or disrupting online services.
This article explains how to assess MySQL table size from table capacity, disk usage, and instance limits, describes why large tables degrade performance due to B+‑tree depth, and presents three practical solutions—partitioning, sharding, and hot‑cold archiving—to improve query speed and manage growth.
This article walks through the full process of designing, configuring, and implementing a sharding solution for large loan and repayment tables using ShardingSphere, Spring Boot 3, MySQL, and MyBatis‑Plus, covering schema planning, data migration, DBA coordination, code examples, common pitfalls, and operational considerations.
This article examines a real‑world MySQL master‑slave replication lag issue caused by massive multi‑table transactions during a sharding operation, details the investigative steps using system tables, binlog parsing, and my2sql analysis, and proposes three practical mitigation strategies.
This article analyzes why large OFFSET values cause severe MySQL performance degradation and presents multiple optimization techniques—including range queries, subqueries, delayed joins, covering indexes, sharding, caching, and search engine integration—along with their advantages, limitations, and practical recommendations.
This article provides a comprehensive guide to Elasticsearch index architecture, covering fundamental concepts, index mapping, field types, alias usage, shard and replica strategies, shard planning, resource impact, and practical recommendations for optimizing performance and stability in production environments.
This article objectively compares middleware‑based sharding with NewSQL distributed databases, examining architecture, distributed transactions, performance, high availability, scaling, SQL support, storage engines, and ecosystem maturity to help architects decide which solution fits their specific workload and operational constraints.
This article systematically examines MySQL optimization techniques for both read‑many/write‑few and write‑many/read‑few business scenarios, covering architecture design, caching, sharding, asynchronous processing, monitoring, and emerging NewSQL/HTAP solutions to achieve high performance, consistency, and scalability.
This article explains the fundamental concepts of database sharding—including data nodes, logical and broadcast tables, sharding keys, strategies, algorithms, SQL parsing, routing, rewriting, execution, result merging, distributed primary keys, data masking, distributed transactions, data migration, and shadow databases—providing clear examples and code snippets for real‑world implementation.
This article analyzes how to build a real‑time, billion‑user leaderboard for a game like Honor of Kings, explaining why traditional ORDER BY queries fail, why Redis sorted sets excel, and detailing sharding, hot‑key mitigation, memory optimization, and fault‑tolerant implementation techniques.
Redis cluster relies on two ports per node, a hash‑slot based sharding scheme, a master‑slave replication model, and provides eventual consistency, making these four fundamentals essential knowledge for developers and operators to.
This article discusses the problem of MySQL auto‑increment ID exhaustion and presents six practical solutions—including changing column type to BIGINT, using UUIDs, segmenting ID generation, composite keys, adjusting auto‑increment steps, and database sharding—to ensure scalability and uniqueness.
This article provides a comprehensive guide to Elasticsearch index concepts, covering index definitions, alias usage, mapping and field types, shard architecture, and practical recommendations for planning shards and optimizing resource consumption to ensure stable and efficient ES clusters.
This article objectively compares middleware‑based sharding with NewSQL distributed databases, examining their architectural differences, transaction models, high‑availability mechanisms, scaling, SQL support, storage engines, and maturity to help practitioners decide which approach best fits their workload and operational constraints.
Although foreign keys enforce referential integrity, many enterprises avoid them because they can slow inserts, cause blocking cascade updates, trigger database update storms in high‑concurrency environments, and complicate sharding, leading developers to prefer redundant fields and handle relationships at the application layer.
The article explains Elasticsearch indexes as logical namespaces for JSON documents, details how mappings define immutable field types such as text, keyword, and numeric, describes primary and replica shard roles, and offers practical guidance on alias usage, shard sizing, replica settings, and performance optimizations to ensure cluster stability.
This article details the end‑to‑end design, configuration, and code implementation of a sharding solution for loan and repayment tables, covering database schema changes, historical data migration, dynamic synchronization switches, transaction handling, and practical pitfalls when using Spring Boot 3 with ShardingSphere.
The case study details how an activity‑middleware platform tackled slow SQL problems by cleaning historic data, introducing sharding and Elasticsearch off‑loading, optimizing queries with proper indexes and simplified joins, and automating nightly deletions, ultimately cutting daily slow queries from thousands to double‑digit levels and boosting system stability.
This article explains why massive tables become slow, identifies common bottlenecks such as disk I/O, missing indexes, deep pagination and lock contention, and provides a step‑by‑step guide covering table design, indexing, SQL tuning, sharding, caching and a real‑world case study to dramatically improve performance.
This article explains Redis's three clustering solutions—master‑slave replication, Sentinel mode, and sharding—detailing their architectures, setup steps, synchronization mechanisms, advantages, drawbacks, split‑brain handling, and common interview questions for developers.
This article compiles the most frequently asked MySQL interview questions, covering fundamentals, storage engines, SQL optimization, indexing, transaction handling, replication, sharding, and performance troubleshooting to help readers master both theory and practical usage.
This article objectively compares NewSQL distributed databases with traditional middleware‑based sharding solutions, examining their architectures, distributed transaction handling, high‑availability, scaling, storage engines, and ecosystem maturity, and provides guidance on selecting the appropriate approach based on consistency, growth, operational capacity, and performance requirements.
This article explains the challenges of storing and serving trillion‑level count data for a massive micro‑blogging platform and presents a step‑by‑step backend design that evolves from simple MySQL tables to sharded databases, Redis caching, message‑queue write‑sharding, and memory‑efficient custom data structures to achieve high performance, low cost, and strong consistency.
This article objectively compares NewSQL databases with middleware‑based sharding, dissecting their core architectures, distributed transaction handling, high‑availability designs, scaling mechanisms, SQL support, storage engines, and maturity to help engineers decide the most suitable solution for their workloads.
An in‑depth comparison of NewSQL databases and middleware‑based sharding reveals each approach’s architectural strengths, distributed transaction handling, scalability, HA mechanisms, storage engine design, and ecosystem maturity, guiding readers on when to adopt NewSQL versus traditional sharding solutions.
This article explains why MongoDB sharding is required for scaling storage and performance, describes the shard, config server, and mongos components, outlines range, hash, and compound sharding strategies, and provides practical guidance on shard key selection, balancing, backup, tuning, and security.
This article explains how to migrate sharded Mycat data to ActionDB with ActionOMS, addressing duplicate primary keys by modifying both source and target schemas, configuring migration tools, and completing validation and post‑migration steps.
A cross‑border e‑commerce company replaced its massive MySQL instance with OceanBase, detailing the business drivers, selection criteria, two‑phase migration process, performance gains, cost savings, remaining incompatibilities, and practical operational lessons learned.
This article examines the core challenges of distributed data systems—including consistency, availability, and partition tolerance—then details master‑slave replication mechanisms for MySQL and Redis, various replication modes and binlog formats, and explores data partitioning, sharding, and hot‑key mitigation techniques for scalable, high‑availability deployments.
This article explores the challenges of handling massive simultaneous requests in distributed architectures and presents practical solutions such as load balancing, distributed caching, asynchronous processing, and sharding, illustrated with case studies from major e‑commerce and social platforms.
This article examines seven typical challenges that arise after implementing database sharding—such as global ID conflicts, cross‑shard queries, distributed transactions, shard‑key design, data migration, pagination, and operational overhead—and provides practical solutions and code examples to address each issue.
This article analyzes the architectural impact of cross‑city multi‑active deployments, focusing on data‑layer design, write latency, sharding strategies, replication topologies, and routing considerations to achieve high availability, performance, and scalability in large‑scale distributed systems.
A ClickHouse cluster showed wildly varying row counts when querying sharded tables, while local tables behaved correctly; the article analyses the root cause in the cluster and table configuration, explains why the inconsistency occurs, and provides a step‑by‑step fix by switching to replicated tables.
When a MySQL table grows to tens of millions of rows, insert and query latency spikes, schema changes become painful, and only a subset of data is needed; this guide explains how to assess table size, why large tables slow down, and three practical solutions—partitioning, sharding, and hot/cold archiving—along with their trade‑offs and implementation steps.
This article explains common data‑migration strategies such as hash‑modulo, time‑range and ID‑range sharding, and compares practical migration solutions including downtime migration, dual‑write, and an MQ + Redis approach, with detailed workflow diagrams and implementation tips.
The article shares a senior architect’s experience with large‑scale MySQL tables, discussing why sharding and Elasticsearch were introduced, the pagination bug caused by Sharding‑JDBC, and a comprehensive set of hardware, software, and SQL tuning recommendations, including a full MySQL configuration example.
This article explains how the Go library BigCache uses sharding, efficient hash functions, and memory‑optimised data structures to deliver high‑concurrency, low‑latency caching, while avoiding GC bottlenecks and providing practical code examples and performance benchmarks.
This article describes how a senior architect designed, configured, and implemented a 50‑table sharding solution for loan and repayment request data using ShardingSphere, Spring Boot 3, MySQL, and custom synchronization scripts, while also addressing historical data migration, backend query changes, and operational safeguards.
This article examines the fundamental drawbacks of MySQL sharding in large‑scale e‑commerce, highlighting distributed transaction challenges, limited cross‑shard queries, global key constraints, scaling difficulties, operational overhead, and why distributed databases may be a better long‑term solution.
This article defines high concurrency, outlines QPS thresholds for small, medium, high, and ultra‑high traffic, and presents practical solutions such as multi‑level caching, load balancing, database sharding, and message‑queue traffic shaping to build robust backend systems.
This article explains why MySQL auto‑increment IDs can run out in large or special‑case tables and presents six practical solutions—including changing the column type to BIGINT, using UUIDs, segmenting IDs, composite keys, adjusting auto‑increment steps, and sharding databases—to ensure scalable and unique primary keys.
This article explains Elasticsearch's distributed architecture, shard and replica model, the detailed write path—including buffer, transaction log, refresh and flush operations—and the read path with coordinating nodes, Lucene query execution, aggregation, caching, and performance optimizations.
This article analyzes how write latency, write volume, isolation, and data replication strategies influence the design of multi‑active distributed systems, offering practical guidance on sharding, synchronous and asynchronous replication, routing, and architecture selection for high availability and performance across regions.
The article walks through how to assess MySQL table size, explains why large tables slow down due to B+‑tree index overhead, and compares three practical solutions—table partitioning, database sharding (horizontal, vertical, hash‑range), and hot‑cold archiving—while outlining their trade‑offs and selection criteria.
MongoDB 8.0 launches with up to 36% higher read throughput, 56% faster batch writes, 200% faster time‑series processing, 50‑fold faster sharding distribution, quantized vector search, and queryable encryption, delivering significant performance, cost, and security improvements for modern workloads.
This article walks through evaluating massive MySQL tables, explains why large row counts slow queries, and compares three practical solutions—table partitioning, database sharding, and hot/cold archiving—while highlighting their trade‑offs and offering guidance on selecting the right approach for a given workload.
This article outlines MongoDB’s core features, its layered technical architecture—including storage, data model, query language, indexing, replication, sharding, and transactions—and explains the three cluster deployment modes with detailed roles and request flow for sharded clusters.
This article objectively compares NewSQL distributed databases with traditional middleware‑based sharding, examining their architectural advantages, distributed transaction handling, high‑availability mechanisms, scaling and sharding strategies, storage engine differences, ecosystem maturity, and provides guidance on selecting the appropriate solution for various workloads.
This article explains how to configure XXL‑JOB with a sharding‑broadcast strategy in a SpringBoot application, use ID modulo hashing to partition large tables, implement the task with XxlJobHelper, write MyBatis sharding SQL, and verify correct parallel processing through detailed logs.
This article explains how to assess MySQL table size using table capacity, disk usage, and instance resources, describes why large tables degrade performance, and presents three practical solutions—partitioning, sharding (horizontal/vertical), and hot‑cold data archiving—along with their trade‑offs and implementation tips.
This article enumerates fifteen practical techniques—including local and distributed caching, parallelism, asynchrony, pooling, sharding, SQL tuning, pre‑computation, transaction trimming, bulk I/O, lock granularity, context propagation, collection sizing, and query optimization—to systematically improve the response time and throughput of service interfaces.
This article introduces MyBatis‑Mate, an enterprise‑grade MyBatis extension that provides sharding, data auditing, sensitive‑word filtering, field encryption, dictionary binding, dynamic DDL maintenance, data‑scope permissions and performance monitoring, complete with practical code examples and configuration guidance.
This technical article describes how a senior architect designed, configured, and migrated loan and repayment application tables to a sharded MySQL setup using ShardingSphere, Spring Boot, and MyBatis‑Plus, covering schema design, historical data sync, dual‑write switches, code integration, and common pitfalls.
MongoDB reported a 13% revenue rise to $478.1 million in Q2 2025, highlighted a major win over PostgreSQL at Fanatics Betting & Gaming, and emphasized its flexible JSON‑like architecture, horizontal scaling, and sharding advantages while noting ongoing competition and a revised growth outlook.
The article introduces mybatis-mate, an enterprise‑grade MyBatis module that supports sharding, multi‑datasource, data auditing, field encryption, sensitive‑data masking, automatic DDL maintenance, performance logging, and fine‑grained data‑scope control, providing code examples and configuration guidelines for Spring Boot projects.
This article objectively compares NewSQL distributed databases with traditional middleware‑based sharding solutions, examining their architectural differences, transaction support, scalability, HA, storage engines, and ecosystem maturity, to help readers decide which approach best fits their performance, consistency, and operational needs.
This article objectively compares NewSQL databases with middleware‑based sharding solutions, examining architecture, distributed transactions, CAP constraints, high availability, scaling, SQL support, storage engines, and maturity to help readers choose the right approach for their workloads.
This article details the design, planning, and step‑by‑step implementation of a database table‑partitioning solution for loan and repayment request tables, covering schema decisions, sharding strategy, historical data migration, dynamic synchronization, code examples with ShardingSphere, SpringBoot, and operational considerations for seamless migration.
This article outlines the progressive evolution of database architectures—from simple monolithic designs to cold‑hot separation, read‑write splitting, and horizontal sharding—explaining each pattern, its components, and the performance and scalability benefits they bring to large‑scale systems.
This article outlines nine practical methods—caching, parallelization, batch processing, data compression, lock‑free design, sharding, request avoidance, pooling, and asynchronous handling—demonstrating how each can be applied to backend services to dramatically reduce latency and improve throughput.
After a year of business coding, the author details the design, implementation, and migration strategy for splitting loan and repayment tables into 50 sharded tables using ShardingSphere with Spring Boot, covering schema design, data synchronization, dynamic switches, and practical pitfalls.
This article explains the core concepts of distributed systems—including the CAP theorem, BASE theory, message‑queue challenges, Redis sentinel issues, sharding strategies, unique ID generation, and distributed transaction patterns—while offering practical guidance to prevent common pitfalls and improve reliability.
This article provides an in‑depth overview of MySQL fundamentals, covering join types, normalization, data types, index structures, locking mechanisms, transaction isolation, MVCC, replication, sharding, and practical performance‑tuning techniques for large‑scale databases.
This article provides a detailed tutorial on database sharding (分库分表), covering concepts, common sharding strategies, a step‑by‑step MyCAT configuration, Java JDBC implementation, and key considerations such as data distribution, complexity, transaction handling, and development cost.
This article objectively compares NewSQL distributed databases with traditional middleware‑based sharding solutions, examining their architectures, transaction models, scalability, high‑availability mechanisms, storage engines, maturity, and offering a decision framework to help engineers choose the most suitable approach for their workloads.
The article outlines practical backend interface performance optimizations—including proper indexing, SQL tuning, parallel remote calls, batch queries, asynchronous processing, scoped transactions, fine-grained locking, pagination batching, multi-level caching, sharding, and monitoring tools—to dramatically reduce latency and improve throughput.
This article introduces MyBatis-Mate, an official MyBatis‑Plus extension offering enterprise‑level capabilities such as dictionary binding, field encryption, data masking, dynamic DDL maintenance, sharding, multi‑datasource switching, performance logging, and data permission control, with detailed Spring Boot integration examples.
This article walks through eleven practical methods—such as adding proper indexes, optimizing SQL, parallelizing remote calls, eliminating duplicate queries, using async processing, shrinking transaction scope, refining lock granularity, applying pagination, leveraging caching layers, and employing sharding and monitoring tools—to dramatically improve backend interface performance.
The article describes how a one‑year‑old loan/repayment service was refactored by introducing a 50‑table sharding scheme with ShardingSphere, detailing the design decisions, historical data migration, code changes, configuration files, pitfalls, and runtime synchronization strategies to achieve efficient query performance on billions of rows.
