BestHub
Sign in
Tagged articles

Sharding

656 articles · Page 1 of 7
samdeepthink
samdeepthink
Aug 18, 2026 · Databases

When a Single Database Hits Its Limits: Sharding Principles and Migration Path

The article explains why sharding should be a last‑resort scaling option, describes vertical and horizontal partitioning, compares hash, range and directory sharding schemes, discusses hotspots, routing, data colocation, migration steps and tool choices, and warns about the operational complexity involved.

Data ConsistencyDatabase ScalingDistributed Databases
0 likes · 16 min read
When a Single Database Hits Its Limits: Sharding Principles and Migration Path
Code Farming
Code Farming
Aug 14, 2026 · Backend Development

How Weibo Handles 100k QPS During Viral Peaks: Architectural Secrets

The article breaks down how Weibo’s backend was engineered to sustain 100,000 queries per second during traffic surges by first quantifying load, then using a push‑pull hybrid feed, a four‑tier cache funnel, and user‑ID sharding with write throttling.

Shardingbackend designcaching
0 likes · 8 min read
How Weibo Handles 100k QPS During Viral Peaks: Architectural Secrets
Cloud Architecture
Cloud Architecture
Aug 12, 2026 · Databases

How Redis Cluster’s Decentralized Design Powers Billion‑Scale Traffic

When a single Redis instance can no longer hold the data volume or write load of e‑commerce workloads, the traditional master‑slave with Sentinel model reaches its limits, and Redis Cluster—by sharding data across 16,384 slots, using gossip‑based topology, and removing a central control plane—delivers horizontal scaling and fault‑tolerance for billions of requests, provided key design, hash tags, hot‑key mitigation, and client routing are applied.

ClusterHigh AvailabilityRedis
0 likes · 32 min read
How Redis Cluster’s Decentralized Design Powers Billion‑Scale Traffic
YiSu Grain
YiSu Grain
Aug 8, 2026 · Databases

Day 51: Database Architecture – Master‑Slave Replication, Read‑Write Separation, Sharding & Consistency

The article walks through diagnosing database bottlenecks, explains MySQL replication flow, read‑write separation benefits and limits, shows when to apply vertical versus horizontal partitioning, details sharding‑key selection, cross‑shard challenges, high‑availability steps, and presents a complete e‑commerce case study.

Database ReplicationHigh AvailabilityHorizontal Partitioning
0 likes · 29 min read
Day 51: Database Architecture – Master‑Slave Replication, Read‑Write Separation, Sharding & Consistency
Xike
Xike
Aug 7, 2026 · Databases

Embedding Sharding Genes in Business IDs for Direct Routing in Sharded Databases

By embedding a shard identifier (“gene”) into the low bits of a business ID generated via Redis INCR or similar sequencers, the article shows how to achieve direct table routing without broadcast queries or extra mapping tables, detailing the algorithm, implementation, integration with Snowflake and Leaf, and common pitfalls.

LeafRedisSharding
0 likes · 11 min read
Embedding Sharding Genes in Business IDs for Direct Routing in Sharded Databases
Ray's Galactic Tech
Ray's Galactic Tech
Aug 1, 2026 · Databases

Beyond CRUD: Full‑Scale Production Guide for MySQL 8.4 LTS

This article walks through a complete production‑grade view of MySQL 8.4 LTS, explaining how a chain of traffic spikes, connection‑pool exhaustion, long transactions and replication lag can cause an avalanche, and then detailing the five core modules, seven production mechanisms, architectural evolution steps, incident post‑mortems, and concrete configuration and code examples to build a resilient MySQL service.

BackupHigh AvailabilityInnoDB
0 likes · 36 min read
Beyond CRUD: Full‑Scale Production Guide for MySQL 8.4 LTS
Code Farming
Code Farming
Jul 29, 2026 · Backend Development

Billions of Logs in Seconds: How Elasticsearch Makes It Possible

The article explains how Elasticsearch achieves sub‑second search over billions of log entries by combining sharding, immutable segment writes, a three‑layer inverted index, and a two‑phase query‑then‑fetch process that distributes work across nodes.

ElasticsearchShardingdistributed search
0 likes · 6 min read
Billions of Logs in Seconds: How Elasticsearch Makes It Possible
Java Architect Handbook
Java Architect Handbook
Jul 26, 2026 · Databases

Interview Question: What Problems Arise After Sharding? List the Key Issues

The article outlines six major challenges introduced by database sharding—cross‑database joins, pagination, distributed transactions, global ID generation, data migration, and aggregation—along with practical solutions such as binding tables, cursor pagination, Seata AT mode, Snowflake IDs, dual‑write scaling, and summary tables, providing interview‑ready answers.

SeataShardingdatabase partitioning
0 likes · 17 min read
Interview Question: What Problems Arise After Sharding? List the Key Issues
Java Architect Handbook
Java Architect Handbook
Jul 24, 2026 · Databases

What Are Database Sharding, Table Sharding, and Combined Sharding? – Xianyu Interview Guide

The article explains the differences between database sharding, table sharding, and combined sharding, outlines interview expectations, details vertical and horizontal splitting strategies, discusses when to apply each technique, and reviews practical tools such as ShardingSphere and MyCat for Java projects.

DatabaseInterview PreparationMySQL
0 likes · 10 min read
What Are Database Sharding, Table Sharding, and Combined Sharding? – Xianyu Interview Guide
Cloud Architecture
Cloud Architecture
Jul 20, 2026 · Databases

Designing MySQL for Millions of QPS: From Single Server to Distributed Architecture

The article walks through a real‑world order system that spikes to 300,000 QPS, explaining why the original single‑node MySQL design fails, and detailing a step‑by‑step evolution—index tuning, transaction fixes, read‑write splitting, vertical and horizontal sharding, plus data‑pipeline integration—to achieve stable low latency at massive scale.

Database ScalingHigh QPSMySQL
0 likes · 20 min read
Designing MySQL for Millions of QPS: From Single Server to Distributed Architecture
Top Architect
Top Architect
Jul 20, 2026 · Databases

Why You Should Stop Using Snowflake for IDs and Try a Shorter MySQL Auto‑Increment Solution

The article examines the drawbacks of using Snowflake for generating short numeric user IDs, details a MySQL auto‑increment based approach, reveals deadlock problems with REPLACE INTO, evaluates alternative schemes, and presents a final sharding‑friendly short‑ID design that meets performance and usability requirements.

DeadlockMySQLSharding
0 likes · 14 min read
Why You Should Stop Using Snowflake for IDs and Try a Shorter MySQL Auto‑Increment Solution
samdeepthink
samdeepthink
Jul 17, 2026 · Databases

Designing Redis Leaderboards: From a Single ZSet to Billion‑Scale Rankings

The article walks through the evolution of Redis leaderboard architectures, covering why ZSets are used, handling same‑score ordering, seasonal resets, access patterns, synchronous vs asynchronous updates, big‑key and hot‑key issues, sharding with buckets, and data durability strategies.

AsynchronousLeaderboardRedis
0 likes · 12 min read
Designing Redis Leaderboards: From a Single ZSet to Billion‑Scale Rankings
Code Farming
Code Farming
Jul 13, 2026 · Backend Development

Designing Trillion‑Scale Counters: From MySQL to a Custom Redis Engine

The article dissects a proven trillion‑level counter architecture, tracing its evolution from a simple MySQL table through hash sharding, a full Redis migration, deep Redis memory optimizations, and hot‑cold separation, while detailing the trade‑offs and performance gains at each step.

Message QueueRedis optimizationSharding
0 likes · 6 min read
Designing Trillion‑Scale Counters: From MySQL to a Custom Redis Engine
IoT Full-Stack Technology
IoT Full-Stack Technology
Jul 13, 2026 · Databases

Is Sharding Dead? A Technical Comparison with NewSQL Databases

The article objectively compares middleware‑based sharding with NewSQL distributed databases, examining distributed transactions, CAP constraints, HA, scaling, storage engines, maturity, and ecosystem, and offers a decision framework for choosing the appropriate architecture based on concrete requirements.

CAP theoremDatabase ArchitectureHigh Availability
0 likes · 18 min read
Is Sharding Dead? A Technical Comparison with NewSQL Databases
Cloud Architecture
Cloud Architecture
Jul 12, 2026 · Databases

Database Performance Optimization: 100× Speed Gains Without Changing SQL

Even without rewriting any SQL, database performance can improve up to a hundredfold by first diagnosing bottlenecks, reducing unnecessary traffic, layering read paths, optimizing indexes, tuning connection pools, and progressively evolving from a single‑node setup to read‑write separation, sharding, and distributed read models.

Connection PoolMySQLRead-Write Separation
0 likes · 38 min read
Database Performance Optimization: 100× Speed Gains Without Changing SQL
Code Farming
Code Farming
Jul 7, 2026 · Backend Development

Four Ways to Ensure Unique IDs After Database Sharding

When a system is split into multiple databases and tables, the article explains the root cause of ID collisions and compares four mainstream ID generation strategies—UUID, database auto‑increment, Snowflake, and segment allocation—detailing their trade‑offs, pitfalls, and selection guidelines for different concurrency levels.

Shardingdistributed IDhigh concurrency
0 likes · 9 min read
Four Ways to Ensure Unique IDs After Database Sharding
Linyb Geek Road
Linyb Geek Road
Jun 30, 2026 · Backend Development

How to Design Pagination for Billion‑Row Sharded Databases in an Interview

The article systematically breaks down pagination challenges in billion‑row sharded databases, compares common sharding strategies and middleware architectures, analyzes the performance drawbacks of a naïve global‑query approach, and presents several practical alternatives—including keyset pagination, two‑stage queries, index‑table tricks, and external search or NewSQL solutions—while highlighting their trade‑offs for interview discussions.

Distributed DatabasesSQLSharding
0 likes · 24 min read
How to Design Pagination for Billion‑Row Sharded Databases in an Interview
Linyb Geek Road
Linyb Geek Road
Jun 30, 2026 · Databases

Implementing Efficient Pagination Across Sharded Databases

The article analyzes why traditional LIMIT/OFFSET pagination fails when data is split across multiple databases, presents a global query approach with its trade‑offs, and proposes an optimized "no‑skip" method plus practical tips using ShardingSphere and Elasticsearch.

DatabaseElasticsearchSharding
0 likes · 7 min read
Implementing Efficient Pagination Across Sharded Databases
Shepherd Advanced Notes
Shepherd Advanced Notes
Jun 24, 2026 · Backend Development

Boosting Throughput 10×: Architecture Evolution and Tuning for High‑Concurrency Batch Processing

The article details how a sluggish batch‑processing system handling millions of records was redesigned with XXL‑JOB sharding, Redis‑based dynamic task distribution, cursor pagination, and selective transaction scopes, achieving nearly ten‑fold throughput improvement while addressing resource contention, load‑balancing, and reliable result reconciliation.

Batch ProcessingMySQLRedis
0 likes · 19 min read
Boosting Throughput 10×: Architecture Evolution and Tuning for High‑Concurrency Batch Processing
samdeepthink
samdeepthink
Jun 22, 2026 · Databases

How to Handle 30,000 Writes per Second with Oracle, Java, and Spring

The article analyzes the write bottleneck of processing 30,000 payment orders per second on an Oracle‑Java‑Spring stack, explains why sharding was used historically, why MQ‑based peak‑shaving is discouraged, and compares self‑built sharding with modern distributed databases such as PolarDB‑X, TiDB and OceanBase, while summarizing public practices from major Chinese tech firms.

Distributed DatabasesMySQLOceanBase
0 likes · 13 min read
How to Handle 30,000 Writes per Second with Oracle, Java, and Spring
Code Farming
Code Farming
Jun 20, 2026 · Backend Development

How This Architecture Handles Tens‑Fold Traffic Spikes Without Crashing

The article breaks down a complete flash‑sale system into four phases and explains how Redis distributed locks, CDN static pages, Nginx rate limiting, message‑queue peak shaving, and sharding together prevent overselling, crashes, and lost orders even when traffic surges dozens of times.

Flash SaleMessage QueueNginx
0 likes · 6 min read
How This Architecture Handles Tens‑Fold Traffic Spikes Without Crashing
Architecture Digest
Architecture Digest
May 15, 2026 · Databases

Why Alibaba Bans Joins Over Three Tables – A Must‑Know Rule for SQL Engineers

Alibaba’s Java Development Manual mandates that any SQL involving more than three tables must be avoided, a rule that stems from the exponential cost of multi‑table joins in a single‑instance database, prompting engineers to rethink data modeling, adopt denormalization, wide tables, materialized views, CQRS or application‑level assembly instead of relying on complex joins.

AlibabaDatabase DesignJOIN
0 likes · 12 min read
Why Alibaba Bans Joins Over Three Tables – A Must‑Know Rule for SQL Engineers
Shuge Unlimited
Shuge Unlimited
Apr 30, 2026 · Databases

Milvus VTS Deep Dive: Two Write Modes, Sharding, and Migration Best Practices

The article provides a source‑code level analysis of Milvus Vector Transport Service (VTS), detailing its three‑stage architecture, partition‑aware sharding logic, two distinct sink write mechanisms (BufferBatchWriter and BulkWriter), schema conversion rules, error‑handling strategies, performance tuning parameters, and practical configuration examples for efficient vector data migration across various data sources.

BufferBatchWriterBulkWriterMilvus
0 likes · 20 min read
Milvus VTS Deep Dive: Two Write Modes, Sharding, and Migration Best Practices
ITPUB
ITPUB
Apr 29, 2026 · Backend Development

How to Rescue a System When QPS Jumps 100× in 10 Minutes

When a service experiences a sudden 100‑fold QPS surge, this guide walks through immediate emergency measures such as rate limiting, circuit breaking, and traffic shedding, followed by systematic analysis of traffic origins, robust architectural redesign including horizontal scaling, micro‑service decomposition, sharding, pooling, caching, and asynchronous processing, and finally stress testing to ensure resilience.

Shardingcachingcircuit breaking
0 likes · 13 min read
How to Rescue a System When QPS Jumps 100× in 10 Minutes
Cloud Architecture
Cloud Architecture
Apr 27, 2026 · Backend Development

Building an Enterprise‑Level MyBatis Persistence Layer from Zero to One

The article walks through a real production incident caused by a massive IN‑list query, then presents a complete methodology for designing, implementing, and tuning an enterprise‑grade MyBatis persistence layer—including core execution chain, caching strategies, batch processing, read/write splitting, sharding, observability, and deployment best practices.

Batch ProcessingMyBatisSQL
0 likes · 39 min read
Building an Enterprise‑Level MyBatis Persistence Layer from Zero to One
Java Companion
Java Companion
Apr 26, 2026 · Databases

Is Sharding Ready to Retire? Why the Classic Split‑Database Approach Is Becoming Legacy

The article reviews the rise and decline of traditional database sharding, explains its technical pitfalls such as wrong shard keys, cross‑database joins, and distributed transactions, compares it with emerging NewSQL solutions like TiDB, OceanBase and PolarDB‑X, and offers practical criteria for choosing the right architecture.

Database ScalingNewSQLOceanBase
0 likes · 14 min read
Is Sharding Ready to Retire? Why the Classic Split‑Database Approach Is Becoming Legacy
Cloud Architecture
Cloud Architecture
Apr 25, 2026 · Databases

How a Phone Number Field Can Crash a 2B‑User Database – Architect’s Design Guide

A mis‑designed phone number column can silently degrade index performance, overload CPU, and cause a cascade of timeouts in a 2‑billion‑user system, but by treating the phone as a domain value object, using a normalized VARCHAR, aligning indexing rules, and applying consistent sharding and migration strategies, you can prevent the database from collapsing under high concurrency.

Database DesignMySQLSchema Migration
0 likes · 31 min read
How a Phone Number Field Can Crash a 2B‑User Database – Architect’s Design Guide
Sohu Tech Products
Sohu Tech Products
Apr 22, 2026 · Databases

How to Resolve ID Conflicts After Sharding? 5 Proven Solutions

When sharding databases, independent auto‑increment IDs can collide, causing data integrity issues; this article explains why the conflict occurs and walks through five practical solutions—including UUID, custom auto‑increment steps, segment mode, Snowflake, and Meituan Leaf—detailing their trade‑offs, performance, and implementation steps.

DatabaseLeafSharding
0 likes · 15 min read
How to Resolve ID Conflicts After Sharding? 5 Proven Solutions
Architecture & Thinking
Architecture & Thinking
Apr 22, 2026 · Backend Development

Why Traditional Limit‑Offset Fails on Billion‑Row Tables and How to Fix It

This article dissects the performance collapse of classic LIMIT‑OFFSET pagination on tables with hundreds of millions of rows, explains the underlying execution steps, presents benchmark data, and walks through a progressive set of solutions—including index tuning, Seek pagination, Cursor pagination, and sharding—complete with Go code examples and practical trade‑offs.

Cursor PaginationMySQLSharding
0 likes · 30 min read
Why Traditional Limit‑Offset Fails on Billion‑Row Tables and How to Fix It
Architect's Guide
Architect's Guide
Apr 18, 2026 · Databases

How to Import 1 Billion Records into MySQL at Lightning Speed

This article analyzes the constraints of loading a billion 1‑KB log records from HDFS/S3 into MySQL, evaluates single‑table limits, proposes batch inserts, sharding, file‑reading techniques, task coordination with Redis, Redisson, and Zookeeper, and offers practical performance‑tuning recommendations.

Bulk InsertMySQLRedis
0 likes · 21 min read
How to Import 1 Billion Records into MySQL at Lightning Speed
Java Companion
Java Companion
Apr 17, 2026 · Backend Development

Resolving ID Conflicts After Sharding: 5 Practical Solutions

When sharding databases, independent auto‑increment IDs can collide; this article explains why the conflict occurs and evaluates five concrete solutions—UUID, MySQL auto‑increment step, segment allocation, Snowflake algorithm, and Meituan Leaf—detailing their mechanisms, trade‑offs, performance, and suitability.

Meituan LeafMySQLSegment
0 likes · 14 min read
Resolving ID Conflicts After Sharding: 5 Practical Solutions
Lobster Programming
Lobster Programming
Apr 15, 2026 · Databases

Choosing the Right Redis Architecture: From Single Node to Cluster

This article reviews the main Redis deployment options—including single‑node, master‑slave with Sentinel, sharding via consistent hashing, and Redis Cluster—explaining their advantages, high‑availability mechanisms, scalability limits, and recommending suitable scenarios for each architecture.

ClusterHigh AvailabilityRedis
0 likes · 7 min read
Choosing the Right Redis Architecture: From Single Node to Cluster
IT Services Circle
IT Services Circle
Apr 11, 2026 · Databases

Why Sharding Isn’t Dead: Modern Alternatives and When to Use Them

The article revisits the rise and fall of database sharding, explains why it became problematic, and evaluates newer cloud‑native, distributed‑SQL, and serverless databases as modern replacements, offering a practical four‑step guide to help engineers choose the right solution for their workload and team.

Cloud NativeDatabase ScalingDistributed SQL
0 likes · 23 min read
Why Sharding Isn’t Dead: Modern Alternatives and When to Use Them
Cloud Architecture
Cloud Architecture
Apr 9, 2026 · Databases

Redis Cluster Deep Dive: Sharding, Replication, Failover & Smart Client

This article provides a comprehensive technical analysis of Redis Cluster, covering its data sharding via 16,384 hash slots, master‑slave replication, automated failover mechanisms, and the essential role of smart clients in routing, along with practical engineering guidelines and production‑grade code examples.

ClusterFailoverRedis
0 likes · 41 min read
Redis Cluster Deep Dive: Sharding, Replication, Failover & Smart Client
Architect's Guide
Architect's Guide
Apr 3, 2026 · Databases

How to Diagnose and Solve MySQL Table Size Bottlenecks

This article explains how to assess MySQL table size using table capacity, disk usage, and instance limits, reveals why large tables slow down due to B+‑tree overhead, and presents three practical solutions—partitioning, sharding, and hot‑cold archiving—to restore query performance.

B+TreeHot Cold ArchivingMySQL
0 likes · 17 min read
How to Diagnose and Solve MySQL Table Size Bottlenecks
Tech Freedom Circle
Tech Freedom Circle
Mar 17, 2026 · Databases

Why HyperLogLog Misses 100M Daily Active Users and How Bitmap Solves It

The article dissects an Alibaba interview question on counting 100 million daily active users, showing why HyperLogLog’s error and lack of per‑user state make it unsuitable, and presents a detailed Bitmap‑based architecture—including sharding, pre‑computation, and ClickHouse integration—to achieve precise, high‑performance analytics.

BitmapClickHouseDailyActiveUsers
0 likes · 16 min read
Why HyperLogLog Misses 100M Daily Active Users and How Bitmap Solves It
Selected Java Interview Questions
Selected Java Interview Questions
Mar 9, 2026 · Databases

How to Paginate Efficiently Across Sharded Databases Without OOM

This article explains why pagination only becomes complex when querying across sharded databases, compares three common solutions—including global search, sequential ID‑based paging, and the discouraged secondary query method—provides practical SQL examples, outlines pros and cons, and offers a STAR‑style interview answer.

SQLShardingpagination
0 likes · 6 min read
How to Paginate Efficiently Across Sharded Databases Without OOM
Senior Xiao Ying
Senior Xiao Ying
Feb 26, 2026 · Databases

Vertical vs Horizontal Sharding in MySQL: Strategy Selection and Middleware Guide

The article explains vertical and horizontal sharding concepts, compares their advantages and drawbacks, evaluates popular sharding middleware such as ShardingSphere, details distributed ID generation with Snowflake, offers cross‑database query solutions, and provides practical guidance on when and how to apply sharding in MySQL.

Cross-Database QueriesHorizontal ShardingMySQL
0 likes · 13 min read
Vertical vs Horizontal Sharding in MySQL: Strategy Selection and Middleware Guide
Architect's Guide
Architect's Guide
Feb 26, 2026 · Backend Development

8 Essential Software Architecture Patterns and When to Use Them

This article explains eight common software architecture patterns—from single‑database apps to microservices, caching, sharding, elastic scaling and multi‑datacenter deployment—detailing their designs, typical use cases, advantages, drawbacks, and practical implementation steps.

Design PatternsElastic ScalingSharding
0 likes · 23 min read
8 Essential Software Architecture Patterns and When to Use Them
IT Learning Made Simple
IT Learning Made Simple
Feb 13, 2026 · Databases

Designing Databases as Intelligent Warehouses: A Supermarket Analogy

This article explains core database design concepts—from relational vs. NoSQL choices and normalization rules to indexing strategies, ACID transactions, SQL optimization, and sharding—using supermarket‑style analogies and concrete exam‑style examples to help readers build fast, safe, and scalable data stores.

Database DesignIndexingSQL Optimization
0 likes · 38 min read
Designing Databases as Intelligent Warehouses: A Supermarket Analogy
Selected Java Interview Questions
Selected Java Interview Questions
Feb 4, 2026 · Databases

How to Handle Pagination Across Sharded Databases Without Performance Pitfalls

This article explains why pagination only becomes complex when queries span multiple database shards, compares three mainstream solutions—including middleware, open‑source frameworks, and business compromises—highlights the pitfalls of global search for deep pages, and recommends sequential paging based on the previous page’s max ID, with interview‑ready STAR guidance.

InterviewSQLSharding
0 likes · 7 min read
How to Handle Pagination Across Sharded Databases Without Performance Pitfalls
Sohu Tech Products
Sohu Tech Products
Jan 28, 2026 · Backend Development

How We Evolved a News App Comment System: From Threaded Views to AI‑Driven Ranking

This article details the evolution of a news‑app comment backend, covering early thread‑based displays, the transition to sharded databases and mixed adjacency‑path models, current hot‑comment ranking strategies, an in‑house experiment platform, topic aggregation via Kafka, and future AI‑driven architectural enhancements.

AIDatabasesRanking
0 likes · 16 min read
How We Evolved a News App Comment System: From Threaded Views to AI‑Driven Ranking
Architect's Guide
Architect's Guide
Jan 27, 2026 · Databases

How to Scale an Order System with Sharding, Distributed IDs, and Seata Transactions

This article details a comprehensive redesign of a high‑traffic order system, covering the challenges of massive data volume, concurrency pressure, and poor scalability, and presenting a step‑by‑step solution that includes sharding strategy selection, database product comparison, unique ID generation, middleware choice, migration plan, risk mitigation, and FAQ handling.

Backend ArchitectureDatabase ScalingSeata
0 likes · 17 min read
How to Scale an Order System with Sharding, Distributed IDs, and Seata Transactions
Architect's Guide
Architect's Guide
Jan 17, 2026 · Backend Development

How We Split a 500M‑Row MySQL Table: Lessons and Strategies

Facing a 50‑million‑row financial transaction table that grew 600,000 rows each month, the team designed a sharding solution using sharding‑jdbc, tackled multi‑datasource transaction and pagination challenges, and executed a staged migration that kept the system stable while eliminating MySQL performance bottlenecks.

ShardingTransactiondata-migration
0 likes · 13 min read
How We Split a 500M‑Row MySQL Table: Lessons and Strategies
Raymond Ops
Raymond Ops
Jan 9, 2026 · Databases

Master MongoDB Sharding: From Single Server to Enterprise-Scale Cluster

When a single‑node MongoDB instance can no longer handle tens of millions of records, this guide walks you through the theory, architecture, deployment steps, shard key strategies, performance tuning, monitoring, backup, and troubleshooting needed to build a robust, production‑grade sharded cluster.

BackupDatabase ScalingMongoDB
0 likes · 14 min read
Master MongoDB Sharding: From Single Server to Enterprise-Scale Cluster
Smart Sea Tide
Smart Sea Tide
Jan 7, 2026 · Databases

Designing Distributed System Architecture with Elasticsearch

The article explains Elasticsearch's cluster architecture, covering nodes, shards, replicas, two deployment models, data‑layer design, and the trade‑offs of each approach, while also comparing alternative distributed storage architectures and their impact on reliability and performance.

Data StorageDeployment ModesDistributed Architecture
0 likes · 14 min read
Designing Distributed System Architecture with Elasticsearch
java1234
java1234
Jan 3, 2026 · Backend Development

Ditch the Heavyweight XXL‑Job: An Elegant Nacos‑Based Scheduling Solution

The article analyses the friction between XXL‑Job and a Nacos‑centric stack, proposes the JobFlow design that removes redundant registration, adds full‑link TraceId, strong sharding with distributed locks, intelligent retries and cloud‑native configuration, and demonstrates how these changes simplify operations and improve observability in microservice environments.

Cloud NativeJobFlowNacos
0 likes · 19 min read
Ditch the Heavyweight XXL‑Job: An Elegant Nacos‑Based Scheduling Solution
Tech Freedom Circle
Tech Freedom Circle
Jan 3, 2026 · Databases

How Many Values Can MySQL IN Handle? 100K‑10M IDs and the Ultimate Solution

An interview scenario reveals that using a massive IN list (hundreds of thousands of IDs) in MySQL quickly exceeds packet limits, forces full‑table scans, and overloads CPU, while batch IN queries cause network storms; the article dissects these pitfalls and presents three progressive strategies—splitting IN, temporary tables with JOIN, and sharding with stream‑merge—to safely query massive ID sets.

IN ClauseMySQLQuery Optimization
0 likes · 32 min read
How Many Values Can MySQL IN Handle? 100K‑10M IDs and the Ultimate Solution
Java Companion
Java Companion
Jan 3, 2026 · Cloud Native

Ditch the Bulky XXL‑Job? Try This Elegant Nacos‑Based Scheduling Solution

The article analyzes the friction between XXL‑Job and Nacos in cloud‑native environments, proposes the JobFlow design that removes redundant registration and configuration, adds full‑traceability, true sharding with distributed locks, smart retries and cloud‑native configuration, and demonstrates how these changes improve consistency, observability and operational cost.

Cloud NativeJobFlowNacos
0 likes · 19 min read
Ditch the Bulky XXL‑Job? Try This Elegant Nacos‑Based Scheduling Solution
Architect
Architect
Dec 28, 2025 · Databases

How to Scale an Order System: Sharding, ID Generation, and Database Choices

This article walks through the challenges of a growing order system, analyzes current bottlenecks, compares sharding and distributed database products, proposes a unique ID strategy, outlines sharding key selection, presents a migration plan with ShardingSphere‑JDBC, and discusses risks and FAQs for a robust backend architecture.

Database ScalingShardingUnique ID
0 likes · 18 min read
How to Scale an Order System: Sharding, ID Generation, and Database Choices
macrozheng
macrozheng
Dec 26, 2025 · Databases

NewSQL vs Middleware Sharding: Which Architecture Really Wins?

This article objectively compares middleware‑based sharding solutions with native NewSQL distributed databases, examining their architectural differences, transaction handling, high‑availability, scaling, SQL support, storage engines, and maturity to help engineers decide which approach best fits their workload.

CAP theoremDistributed DatabasesNewSQL
0 likes · 20 min read
NewSQL vs Middleware Sharding: Which Architecture Really Wins?
Architect Chen
Architect Chen
Dec 23, 2025 · Databases

Boost MySQL to 10k QPS: Multi‑Layer Optimization Strategies

When MySQL QPS reaches 10,000, common bottlenecks include CPU saturation, low buffer pool hit rate, connection limits, and lock contention, which can be mitigated through internal tuning, cache interception, read‑write splitting, and horizontal sharding.

MySQLOptimizationQPS
0 likes · 4 min read
Boost MySQL to 10k QPS: Multi‑Layer Optimization Strategies
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 21, 2025 · Backend Development

How Elasticsearch Scales to Billions of Queries: Sharding, Inverted Index, Distributed Execution, and Replication

Elasticsearch achieves billion‑scale search performance by combining horizontal sharding, immutable inverted‑index segments, a two‑stage distributed Query/FETCH model, and multiple replicas with a coordinator node to ensure high concurrency, scalability, and availability.

Distributed QueryElasticsearchSharding
0 likes · 4 min read
How Elasticsearch Scales to Billions of Queries: Sharding, Inverted Index, Distributed Execution, and Replication
Su San Talks Tech
Su San Talks Tech
Dec 11, 2025 · Databases

How to Scale a 900 M‑Row Message Table to 60 B Rows Without Downtime

This article details a comprehensive sharding strategy for a high‑traffic message table, explaining why early partitioning is essential, how to design language‑based vertical shards combined with hash‑based horizontal shards, calculate table counts and virtual nodes, and execute a zero‑downtime migration using dual‑writes and gradual traffic rollout.

Horizontal PartitioningMySQLSharding
0 likes · 16 min read
How to Scale a 900 M‑Row Message Table to 60 B Rows Without Downtime
Java Architect Handbook
Java Architect Handbook
Nov 30, 2025 · Databases

When Sharding Fails: Debugging Pagination Bugs and Optimizing MySQL for Massive Tables

The article recounts a real‑world case where a massive device table (over 500 million rows) suffered from sharding and Elasticsearch integration issues, describes a pagination bug caused by Sharding‑JDBC, and presents a comprehensive set of hardware, software, and SQL optimization techniques—including MySQL configuration examples—to improve performance.

ElasticsearchMySQLSharding
0 likes · 13 min read
When Sharding Fails: Debugging Pagination Bugs and Optimizing MySQL for Massive Tables
Code Wrench
Code Wrench
Nov 27, 2025 · Databases

Build a Mini Olric KV Store in Go: 300 Lines of Sharding, TTL, and Performance Tuning

This article walks through implementing a compact, 300‑line Go version of Olric—a distributed key‑value store—covering core data structures, shard routing, simplified RPC, TTL handling, node replication, rebalancing, concurrency safety, and performance experiments with benchmarks, profiling, and memory optimizations.

Distributed KVGoOlric
0 likes · 9 min read
Build a Mini Olric KV Store in Go: 300 Lines of Sharding, TTL, and Performance Tuning
Java Companion
Java Companion
Nov 21, 2025 · Databases

How to Query Phone Number Suffix in Milliseconds on 50 Million Records

When faced with 50 million user rows, using LIKE '%1234' triggers a full table scan, but adding a suffix column, reversing the phone number, leveraging function indexes, or integrating Elasticsearch can reduce query time from minutes to a few milliseconds, each with its own trade‑offs.

Database DesignElasticsearchIndexing
0 likes · 12 min read
How to Query Phone Number Suffix in Milliseconds on 50 Million Records
Architect-Kip
Architect-Kip
Nov 21, 2025 · Backend Development

Mastering Pagination Performance: Single‑DB and Cross‑DB Strategies

This guide explores common pagination bottlenecks and presents practical solutions for single‑database and sharded environments, covering keyset pagination, bidirectional paging, join‑based paging, index design, middleware rewriting, approximate paging, and two‑phase query techniques.

Keyset PaginationMiddlewareSQL
0 likes · 14 min read
Mastering Pagination Performance: Single‑DB and Cross‑DB Strategies
Architect Chen
Architect Chen
Nov 18, 2025 · Backend Development

Designing High‑Concurrency Microservices: Splitting, Sharding, and Resilience

This guide explains how to break a monolithic system into independent microservices, apply vertical and horizontal data sharding, and implement rate limiting, circuit breaking, and degradation to sustain millions of concurrent requests while keeping the backend stable and scalable.

Shardingcircuit breakingmicroservices
0 likes · 5 min read
Designing High‑Concurrency Microservices: Splitting, Sharding, and Resilience
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Nov 13, 2025 · Backend Development

Designing High‑Concurrency Microservice Architectures: Splitting, Sharding, Rate Limiting, and Circuit Breaking

This guide explains how to build a million‑request‑per‑second microservice system by properly splitting business domains, partitioning data with vertical and horizontal sharding, applying robust rate‑limiting techniques, and implementing circuit‑breaking and degradation strategies to maintain stability.

Service SplittingShardingcircuit breaking
0 likes · 6 min read
Designing High‑Concurrency Microservice Architectures: Splitting, Sharding, Rate Limiting, and Circuit Breaking
Selected Java Interview Questions
Selected Java Interview Questions
Nov 6, 2025 · Databases

Master Spring Boot ShardingSphere: Step-by-Step MySQL Sharding Guide

This article walks through building a Spring Boot project with ShardingSphere‑JDBC, covering Maven dependencies, YAML configuration, database setup, sharding rules, distributed ID generation, binding tables, and testing, providing a complete, hands‑on guide to implementing MySQL horizontal partitioning in a real‑world application.

MySQLShardingShardingSphere
0 likes · 18 min read
Master Spring Boot ShardingSphere: Step-by-Step MySQL Sharding Guide
Senior Brother's Insights
Senior Brother's Insights
Oct 29, 2025 · Databases

Sharding vs Partitioning: When and How to Scale Your Database

This article explains the concepts, differences, advantages, and disadvantages of database sharding and partitioning, provides practical MySQL examples for horizontal sharding by region and range partitioning, and offers guidance on choosing the right approach for scalability and maintainability.

Data DistributionDatabase ScalingMySQL
0 likes · 10 min read
Sharding vs Partitioning: When and How to Scale Your Database
Tech Freedom Circle
Tech Freedom Circle
Oct 25, 2025 · Databases

Designing a 10 WQPS Redis Counter Component: A Systematic Timer Solution

This article presents a complete, step‑by‑step analysis of a high‑concurrency Redis counter component that supports up to 100 000 QPS, covering business pain points, architectural design, two core counting strategies, sharding, local batch optimization, code walkthroughs, and performance benchmark results.

AOPCounterRedis
0 likes · 33 min read
Designing a 10 WQPS Redis Counter Component: A Systematic Timer Solution
Tech Freedom Circle
Tech Freedom Circle
Oct 23, 2025 · Databases

Why Consistent Hashing Fails: Why Redis, HBase, TiDB and Ceph Have Dropped It

The article examines the fundamental limitations of consistent hashing—its inability to preserve data locality, support range queries, and handle topology awareness—explaining why major storage systems such as Redis Cluster, TiDB, Ceph, and HBase have adopted alternative sharding strategies like hash slots, range partitioning, and CRUSH.

CRUSHCephHBase
0 likes · 45 min read
Why Consistent Hashing Fails: Why Redis, HBase, TiDB and Ceph Have Dropped It
Senior Brother's Insights
Senior Brother's Insights
Oct 15, 2025 · Databases

Master MySQL Partitioning: When, How, and Why It Beats Sharding

This guide explains MySQL partitioning fundamentals, shows how to create RANGE, LIST, and KEY partitions with concrete CREATE TABLE examples, compares the performance benefits and maintenance trade‑offs, and clarifies the differences between partitioning and sharding to help you decide when to use each technique.

KEYMySQLPartitioning
0 likes · 7 min read
Master MySQL Partitioning: When, How, and Why It Beats Sharding
Su San Talks Tech
Su San Talks Tech
Oct 9, 2025 · Backend Development

Designing a Scalable Real‑Time Video Ranking System for Billions of Views

This article walks through the end‑to‑end system design of a TikTok‑style video leaderboard, covering functional and non‑functional requirements, traffic estimation, single‑node prototypes, fault‑tolerant replication, sharding, elastic partitioning, sliding‑window counters, and caching strategies to achieve low‑latency, high‑throughput ranking at massive scale.

Distributed ArchitectureShardingSliding Window
0 likes · 21 min read
Designing a Scalable Real‑Time Video Ranking System for Billions of Views
Su San Talks Tech
Su San Talks Tech
Sep 28, 2025 · Databases

5 Redis Deployment Patterns Every Engineer Should Know

The article outlines five Redis deployment patterns—from a single‑instance setup to official Redis Cluster—detailing their architectures, advantages, drawbacks, and suitable scenarios, helping engineers choose the right solution for scalability, high availability, and operational complexity.

Consistent HashRedis ClusterSharding
0 likes · 8 min read
5 Redis Deployment Patterns Every Engineer Should Know
Raymond Ops
Raymond Ops
Sep 22, 2025 · Databases

Master‑Slave, Sentinel, and Sharding: Complete Guide to Redis Cluster Architectures

This article explains Redis’s three clustering options—master‑slave replication, Sentinel high‑availability, and sharding—detailing their architectures, setup steps, synchronization mechanisms, advantages, drawbacks, and common interview questions, helping readers choose and implement the right solution for high‑performance, scalable data storage.

ClusterHigh AvailabilityRedis
0 likes · 18 min read
Master‑Slave, Sentinel, and Sharding: Complete Guide to Redis Cluster Architectures
Su San Talks Tech
Su San Talks Tech
Sep 22, 2025 · Backend Development

Mastering Pagination in Billion‑Row Sharded Databases: Interview‑Ready Strategies

This article dissects the challenges of implementing pagination on billion‑row sharded tables, explains common sharding strategies, compares SDK, Proxy, and Sidecar architectures, and presents practical solutions such as global query, infinite scroll, two‑phase queries, index tables, and external storage to help engineers ace interview questions.

Distributed DatabasesShardingbackend interview
0 likes · 24 min read
Mastering Pagination in Billion‑Row Sharded Databases: Interview‑Ready Strategies
Code Wrench
Code Wrench
Sep 17, 2025 · Databases

Mastering GORM Sharding: A Lightweight Go Plugin for Scalable Databases

This article explains why horizontal sharding becomes essential for billion‑row tables, introduces the non‑intrusive GORM Sharding plugin, walks through installation, configuration, and code examples, and evaluates its advantages, limitations, common issues, and suitable use cases for Go back‑end services.

GORMGoSharding
0 likes · 11 min read
Mastering GORM Sharding: A Lightweight Go Plugin for Scalable Databases
Architect's Journey
Architect's Journey
Sep 10, 2025 · Databases

Why Taobao Order IDs End with the Same Six Digits: Inside the Gene Method

The article explains why the last six digits of Taobao order numbers are constant, revealing the “gene” method that embeds a hashed user identifier to bind orders to specific shards, thereby improving query performance, ensuring balanced data distribution, and supporting scalability in large e‑commerce systems.

Database DesignDistributed DatabasesSharding
0 likes · 8 min read
Why Taobao Order IDs End with the Same Six Digits: Inside the Gene Method
Java Backend Full-Stack
Java Backend Full-Stack
Sep 9, 2025 · Backend Development

Why Do the Last Six Digits of Taobao Order IDs Remain Constant?

The article explains how Taobao embeds a user‑specific “gene” in the last six digits of its order numbers, enabling efficient sharding routing, uniform data distribution, and idempotent order handling while maintaining global uniqueness in a large e‑commerce system.

ShardingTaobaodatabase routing
0 likes · 7 min read
Why Do the Last Six Digits of Taobao Order IDs Remain Constant?
JD Cloud Developers
JD Cloud Developers
Sep 2, 2025 · Databases

Unlocking ClickHouse’s Lightning‑Fast Queries: The ‘Nine Swords’ Architecture Explained

This article explores ClickHouse’s high‑performance OLAP design—including its MPP architecture, columnar storage, vectorized execution, pre‑sorting, sharding, replication, index strategies, and compute engine—showing how each innovation contributes to ultra‑fast, scalable data analysis in the big‑data era.

ClickHouseShardingcolumnar storage
0 likes · 14 min read
Unlocking ClickHouse’s Lightning‑Fast Queries: The ‘Nine Swords’ Architecture Explained
Raymond Ops
Raymond Ops
Aug 22, 2025 · Operations

Mastering ELK Stack: From Installation to Advanced Sharding Strategies

This guide introduces the ELK stack fundamentals, explains Elasticsearch, Logstash, and Kibana roles, walks through environment preparation, installation, configuration, head plugin setup, shard and replica concepts, scaling recommendations, and provides scripts for monitoring cluster health, offering a comprehensive hands‑on reference for log analytics operations.

ELKElasticsearchKibana
0 likes · 16 min read
Mastering ELK Stack: From Installation to Advanced Sharding Strategies
MaGe Linux Operations
MaGe Linux Operations
Aug 12, 2025 · Databases

Master MongoDB Sharding: From Single Server to Scalable Cluster Deployment

This comprehensive guide explains MongoDB sharding fundamentals, walks through step‑by‑step deployment of config servers, shard replica sets, and mongos routers, compares range, hash, and compound shard keys, and provides performance tuning, security, backup, monitoring, and troubleshooting best practices for production‑grade clusters.

Cluster DeploymentConfig ServersDatabase Scaling
0 likes · 13 min read
Master MongoDB Sharding: From Single Server to Scalable Cluster Deployment
Tech Freedom Circle
Tech Freedom Circle
Aug 4, 2025 · Operations

How Do Projects Achieve High Availability Without Multi‑Site Active‑Active? – A Meituan Interview Question

The article analyzes high‑availability concepts, from single‑machine risks to multi‑site active‑active architectures, compares cold and hot backup strategies, discusses network latency challenges, and presents Ele.me’s cell‑based, sharding‑driven multi‑region solution with concrete examples, tables, and code snippets.

Data ReplicationDisaster RecoveryHigh Availability
0 likes · 28 min read
How Do Projects Achieve High Availability Without Multi‑Site Active‑Active? – A Meituan Interview Question
MaGe Linux Operations
MaGe Linux Operations
Aug 1, 2025 · Databases

Master MySQL Slow Query Fixes: From Index Tuning to Distributed Sharding

This comprehensive guide walks MySQL administrators through diagnosing slow queries, configuring the slow‑query log, leveraging EXPLAIN, optimizing indexes, rewriting joins, implementing pagination tricks, and scaling with read‑write splitting, master‑slave replication, and horizontal sharding to dramatically improve performance in high‑traffic production environments.

MySQLShardingindex optimization
0 likes · 21 min read
Master MySQL Slow Query Fixes: From Index Tuning to Distributed Sharding
Top Architect
Top Architect
Jul 24, 2025 · Databases

How to Efficiently Shard Billion‑Row Tables with ShardingSphere and Spring Boot

This article walks through the end‑to‑end design, configuration, and implementation of splitting massive loan and repayment tables into 50 sharded MySQL tables using ShardingSphere, Spring Boot, and a custom suffix algorithm, while covering data migration, DBA coordination, dynamic switches, and scheduled consistency checks.

Data MigrationMySQLSharding
0 likes · 19 min read
How to Efficiently Shard Billion‑Row Tables with ShardingSphere and Spring Boot
ITPUB
ITPUB
Jul 14, 2025 · Databases

How to Compute Mutual Friends for 100M Users in Seconds with Redis

This article breaks down the massive‑scale challenge of calculating mutual friends for hundreds of millions of users, analyzes data‑size, latency, and memory constraints, and presents three Redis‑based solutions—basic Set, bitmap, and a sharded architecture—complete with code snippets, performance numbers, and practical trade‑offs.

BitmapMutual FriendsRedis
0 likes · 10 min read
How to Compute Mutual Friends for 100M Users in Seconds with Redis
Tech Freedom Circle
Tech Freedom Circle
Jul 14, 2025 · Databases

How to Estimate Sharding Capacity: Calculating Required Databases and Tables for an Alibaba Interview

The article walks through why sharding is needed, outlines IO and CPU bottlenecks, presents two design principles, shows how to estimate capacity from existing data and growth trends, compares range, modulo, consistent‑hash and Snowflake sharding schemes, and details migration strategies for expanding nodes without downtime.

Data MigrationShardingcapacity planning
0 likes · 22 min read
How to Estimate Sharding Capacity: Calculating Required Databases and Tables for an Alibaba Interview