BestHub
Sign in
Tagged articles

Sharding

656 articles · Page 3 of 7
Architecture Digest
Architecture Digest
Apr 19, 2024 · Databases

Comparing NewSQL Distributed Databases with Middleware‑Based Sharding: Advantages, Trade‑offs, and Use Cases

The article objectively compares NewSQL distributed databases with traditional middleware‑based sharding solutions, examining their architectural differences, distributed transaction support, performance, scalability, high‑availability mechanisms, storage engines, and practical suitability for various application scenarios.

CAP theoremDistributed DatabasesHigh Availability
0 likes · 18 min read
Comparing NewSQL Distributed Databases with Middleware‑Based Sharding: Advantages, Trade‑offs, and Use Cases
Top Architect
Top Architect
Apr 18, 2024 · Big Data

Understanding ElasticSearch Architecture and Its Underlying Lucene Mechanics

This article provides a comprehensive, top‑down and bottom‑up explanation of ElasticSearch’s core architecture, detailing nodes, shards, Lucene segments, inverted indexes, stored fields, document values, caching, query processing, routing, and scaling considerations for efficient search operations.

LuceneShardinginverted index
0 likes · 10 min read
Understanding ElasticSearch Architecture and Its Underlying Lucene Mechanics
Architect
Architect
Apr 17, 2024 · Databases

How We Split a 500‑Million‑Row MySQL Table: Process, Pitfalls, and Lessons

Facing a 50‑million‑row financial transaction table that grew 6 million rows per month, the team analyzed the problem, set clear split goals, evaluated sharding middleware, designed a custom pagination algorithm, built a hybrid data‑migration plan, and executed a three‑stage rollout to safely replace the monolithic table with multiple 10‑million‑row shards.

Database MigrationLarge TablesMySQL
0 likes · 13 min read
How We Split a 500‑Million‑Row MySQL Table: Process, Pitfalls, and Lessons
FunTester
FunTester
Mar 21, 2024 · Operations

How Dynamic Task‑Grabbing Cuts Distributed Batch Jobs from Hours to Minutes

This article presents a detailed case study of optimizing a distributed batch processing system by replacing static shard‑key concurrency with a dynamic task‑grabbing mechanism, dramatically reducing execution time from several hours to under fifteen minutes while maintaining stable resource usage.

Batch ProcessingLoad BalancingSharding
0 likes · 8 min read
How Dynamic Task‑Grabbing Cuts Distributed Batch Jobs from Hours to Minutes
Ctrip Technology
Ctrip Technology
Mar 8, 2024 · Backend Development

Design and Implementation of Ctrip Flight Order Itinerary System: Architecture, Sharding, Caching, and Structured Concurrency

The article presents a comprehensive case study of Ctrip's flight order itinerary system, detailing its background challenges, design goals, modular architecture, data aggregation strategies, sharding and Redis caching optimizations, and the adoption of Java structured concurrency to achieve high performance, reliability, and scalability.

ShardingStructured Concurrencybackend
0 likes · 13 min read
Design and Implementation of Ctrip Flight Order Itinerary System: Architecture, Sharding, Caching, and Structured Concurrency
Architecture & Thinking
Architecture & Thinking
Mar 5, 2024 · Databases

How Database Middleware Solves High‑Traffic Challenges: Connection Pools, Sharding, and More

This article examines how database middleware tackles the demanding needs of large‑scale internet services by providing centralized connection‑pool management, transparent read‑write splitting, diverse load‑balancing algorithms, sharding support, automatic failover, security controls, comprehensive monitoring, and flexible backup‑recovery mechanisms.

Connection PoolFault ToleranceMonitoring
0 likes · 9 min read
How Database Middleware Solves High‑Traffic Challenges: Connection Pools, Sharding, and More
21CTO
21CTO
Mar 1, 2024 · Databases

Why Oracle 23c’s Global Distributed Database Is a Game-Changer for Multi‑Region Data

The article examines Oracle’s evolution from RAC‑based sharding to the 23c Global Distributed Database, highlighting its RAFT replication, cross‑region capabilities, compliance benefits, and performance advantages that position it as a true multi‑cloud, multi‑location solution for modern OLTP workloads.

OracleRaftSharding
0 likes · 7 min read
Why Oracle 23c’s Global Distributed Database Is a Game-Changer for Multi‑Region Data
JD Retail Technology
JD Retail Technology
Feb 29, 2024 · Databases

Optimizing Large‑Scale Batch Processing for an Advertising Platform: From Query Tuning to Load‑Balanced Execution

This article presents a real‑world case study of optimizing massive batch‑processing tasks in an ad‑platform by applying query‑level improvements, cursor‑based pagination, shard‑aware batch updates, JVM‑tuned garbage collection, and distributed load‑balancing, ultimately reducing CPU usage from 80% to under 2% and cutting query‑per‑minute volume from millions to a few thousand.

Batch ProcessingCursor PaginationSharding
0 likes · 22 min read
Optimizing Large‑Scale Batch Processing for an Advertising Platform: From Query Tuning to Load‑Balanced Execution
JD Tech
JD Tech
Feb 19, 2024 · Databases

Choosing and Generating Sharding Keys for Database Partitioning

This article explains how to select appropriate sharding keys based on business requirements and data access patterns, describes vertical and horizontal partitioning methods, and presents common techniques such as equal‑value, index, and gene methods along with key generation strategies like auto‑increment, UUID, Snowflake algorithm, and open‑source middleware.

DatabaseHorizontal PartitionSharding
0 likes · 10 min read
Choosing and Generating Sharding Keys for Database Partitioning
Architect
Architect
Feb 17, 2024 · Backend Development

How Bilibili Scaled Its Membership Purchase System: Call‑Chain Refactor, Async Ordering, and Sharding

This article details how Bilibili’s membership‑purchase platform tackled massive traffic spikes by redesigning the order call chain, introducing concurrent and asynchronous processing, and applying a sharding strategy that split databases and tables, ultimately boosting latency performance and supporting over 4,000 TPS during peak sales.

Async ProcessingDatabase ScalingSharding
0 likes · 15 min read
How Bilibili Scaled Its Membership Purchase System: Call‑Chain Refactor, Async Ordering, and Sharding
ITPUB
ITPUB
Feb 13, 2024 · Databases

Achieve Seamless Second‑Level Database Scaling for High‑Throughput Microservices

This guide explains how to design a high‑concurrency, high‑throughput internet architecture that ensures database high availability with double‑master sync and virtual IPs, and how to horizontally shard and smoothly expand the cluster in seconds using configuration changes, reloads, and cleanup steps.

DatabasesHigh AvailabilityScaling
0 likes · 8 min read
Achieve Seamless Second‑Level Database Scaling for High‑Throughput Microservices
JavaEdge
JavaEdge
Feb 7, 2024 · Backend Development

Designing a High‑Availability Payment System: Flow, Optimization, and Fault Tolerance

This article details the end‑to‑end design of a payment system, covering transaction flow, horizontal and vertical pre‑optimizations, task scheduling, sharding strategies, data structures, high‑availability mechanisms such as channel isolation and Hystrix, and future planning for dynamic scaling and intelligent routing.

Backend ArchitectureElastic-JobHigh Availability
0 likes · 12 min read
Designing a High‑Availability Payment System: Flow, Optimization, and Fault Tolerance
Architect's Guide
Architect's Guide
Feb 4, 2024 · Databases

Evaluating Large MySQL Table Size and Optimizing Query Performance with Partitioning, Sharding, and Archiving

This article explains how to assess MySQL table size through table capacity, disk usage, and instance resources, illustrates the impact of B+‑tree index depth on query speed, and presents three practical solutions—table partitioning, database sharding, and hot‑cold data archiving—to improve performance.

Data ArchivingMySQLPartitioning
0 likes · 15 min read
Evaluating Large MySQL Table Size and Optimizing Query Performance with Partitioning, Sharding, and Archiving
Bilibili Tech
Bilibili Tech
Feb 2, 2024 · Backend Development

Performance Optimization of Bilibili Membership Purchase Transaction System

Bilibili’s membership purchase system was re‑engineered by refactoring serial service calls into a responsibility‑chain with parallelism, moving weak dependencies to asynchronous queues, and implementing sharding across 256 tables, boosting peak throughput from ~600 QPS to over 4,000 TPS and eliminating latency incidents during massive promotional traffic spikes.

Shardingasynchronous orderbackend development
0 likes · 13 min read
Performance Optimization of Bilibili Membership Purchase Transaction System
Efficient Ops
Efficient Ops
Jan 8, 2024 · Databases

Eight Proven Strategies to Supercharge Database Performance

This article outlines why databases become slow, introduces a four‑layer optimization framework, and presents eight practical solutions—including data reduction, archiving, sharding, caching, and read‑replica strategies—to help engineers systematically diagnose and resolve the majority of performance bottlenecks.

Data ArchivingNoSQLSharding
0 likes · 20 min read
Eight Proven Strategies to Supercharge Database Performance
macrozheng
macrozheng
Dec 28, 2023 · Databases

Mastering Sharding: 21 Core Concepts for Scalable Database Design

This article introduces the essential concepts of sharding architecture—including data partitioning, logical and physical tables, sharding keys, routing strategies, SQL rewriting, distributed primary keys, data masking, transactions, migration, and shadow databases—providing a comprehensive guide for building high‑performance, horizontally scaled database systems.

Broadcast TableSQL RoutingSharding
0 likes · 21 min read
Mastering Sharding: 21 Core Concepts for Scalable Database Design
Code Ape Tech Column
Code Ape Tech Column
Dec 26, 2023 · Databases

Eight Major Database Optimization Strategies for Backend Engineers

This article outlines eight comprehensive database optimization approaches—reducing data volume, using space for performance, selecting appropriate storage systems, and related techniques such as data archiving, intermediate tables, serialization, sharding, distributed caching, read‑write splitting, and CQRS—to help backend engineers systematically address common performance bottlenecks.

Shardingbackend engineeringcqrs
0 likes · 23 min read
Eight Major Database Optimization Strategies for Backend Engineers
dbaplus Community
dbaplus Community
Dec 25, 2023 · Databases

When Should You Choose Distributed Over Centralized Databases? A Practical Guide

This article examines the current landscape of Chinese databases, compares centralized and distributed architectures, outlines when distributed solutions are truly needed, provides performance test data, and offers practical advice on sharding, SQL design, and avoiding cross‑node bottlenecks.

Database ArchitectureDistributed DatabasesSharding
0 likes · 14 min read
When Should You Choose Distributed Over Centralized Databases? A Practical Guide
IT Niuke
IT Niuke
Dec 23, 2023 · Backend Development

Mastering Distributed Scheduling with XXL‑JOB: A Hands‑On Tutorial

This article explains the fundamentals of scheduled tasks, highlights the limitations of single‑node solutions, introduces the open‑source XXL‑JOB platform, walks through its architecture, installation, configuration, task creation, routing strategies, clustering, sharding and GLUE mode, and offers practical code examples for Java/Spring‑Boot projects.

ClusterShardingSpring Boot
0 likes · 26 min read
Mastering Distributed Scheduling with XXL‑JOB: A Hands‑On Tutorial
Nullbody Notes
Nullbody Notes
Dec 12, 2023 · Backend Development

Build a High‑Performance Go Cache Library (EasyCache) from Scratch

This article walks through implementing EasyCache, a Go‑based in‑memory cache with sharding, lock‑free concurrency, LRU eviction, and configurable expiration, explaining the underlying data structures, goroutine cleanup logic, and key handling with concrete code examples.

GoLRUSharding
0 likes · 9 min read
Build a High‑Performance Go Cache Library (EasyCache) from Scratch
Selected Java Interview Questions
Selected Java Interview Questions
Nov 30, 2023 · Databases

Designing a High‑Performance System to Import 1 Billion Records into MySQL

To import one billion 1 KB log records stored in HDFS or S3 into MySQL as quickly as possible, the article analyzes constraints, evaluates single‑table limits, proposes sharding, batch inserts, storage‑engine choices, file‑reading strategies, task coordination, reliability mechanisms, and concurrency control.

Shardingdata importperformance optimization
0 likes · 19 min read
Designing a High‑Performance System to Import 1 Billion Records into MySQL
Aikesheng Open Source Community
Aikesheng Open Source Community
Nov 29, 2023 · Databases

When to Use Distributed vs. Centralized Databases: Analysis, Benchmarks, and Best Practices

This article examines the trade‑offs between centralized and distributed OLTP databases, presents industry usage statistics, performance benchmarks, practical questions for migration, and detailed guidance on sharding, SQL design, and operational considerations to help decide when a distributed solution is truly needed.

Database ArchitectureDistributed DatabasesOLTP
0 likes · 12 min read
When to Use Distributed vs. Centralized Databases: Analysis, Benchmarks, and Best Practices
Top Architecture Tech Stack
Top Architecture Tech Stack
Nov 20, 2023 · Databases

Design and Application of High‑Performance Database Clusters: Read/Write Separation, Sharding, and NoSQL

This article explains the principles and practical designs of high‑performance database clusters, covering read/write separation, master‑slave replication lag, distribution mechanisms, sharding (both vertical and horizontal), business‑level partitioning, and the role of NoSQL technologies such as key‑value stores, document databases, columnar databases, and full‑text search engines.

DatabaseNoSQLRead-Write Separation
0 likes · 33 min read
Design and Application of High‑Performance Database Clusters: Read/Write Separation, Sharding, and NoSQL
Architect's Guide
Architect's Guide
Nov 15, 2023 · Databases

Smooth 2N Database Scaling and High Availability with MariaDB, Keepalived, and Sharding

This article presents five expansion strategies—shutdown, write‑stop, log‑based, dual‑write, and smooth 2N—detailing step‑by‑step procedures for MariaDB installation, master‑master replication, dynamic data‑source configuration, and Keepalived high‑availability setup, enabling seamless horizontal scaling and minimal service disruption for large‑scale databases.

Database ScalingHigh AvailabilityKeepalived
0 likes · 30 min read
Smooth 2N Database Scaling and High Availability with MariaDB, Keepalived, and Sharding
IT Services Circle
IT Services Circle
Nov 12, 2023 · Databases

Why Many Large Internet Companies Avoid Using MySQL Foreign Keys

The article explains that while MySQL foreign keys ensure data consistency and integrity, they introduce performance overhead, lock contention, and scalability issues—especially in high‑concurrency and sharded environments—leading many large internet firms to forego them in favor of application‑level solutions.

Database PerformanceForeign KeyMySQL
0 likes · 5 min read
Why Many Large Internet Companies Avoid Using MySQL Foreign Keys
Top Architecture Tech Stack
Top Architecture Tech Stack
Nov 2, 2023 · Databases

Strategies for Efficiently Importing One Billion Records into MySQL

This article analyzes the constraints of loading one billion 1 KB log records from distributed storage into MySQL, evaluates single‑table limits, proposes batch‑insert, sharding, storage‑engine, file‑reading, and distributed‑task coordination techniques to achieve high‑speed, ordered, and reliable data ingestion.

Batch InsertMySQLRedis
0 likes · 18 min read
Strategies for Efficiently Importing One Billion Records into MySQL
Java High-Performance Architecture
Java High-Performance Architecture
Nov 2, 2023 · Databases

How to Tackle Large MySQL Tables: Partitioning, Sharding, and Archiving Strategies

When MySQL tables grow to millions of rows, insert and query latency increase, schema changes become costly, and irrelevant data bloats storage, so this guide evaluates table size, explains why B+‑tree depth hurts performance, and presents partitioning, sharding, and hot‑cold archiving solutions with practical SQL examples.

B+TreeMySQLSharding
0 likes · 15 min read
How to Tackle Large MySQL Tables: Partitioning, Sharding, and Archiving Strategies
Architect
Architect
Oct 29, 2023 · Databases

MongoDB Cluster Architecture: Master‑Slave, Replica Set, and Sharding

This article explains MongoDB's three cluster architectures—deprecated Master‑Slave replication, the highly available Replica Set with automatic failover, and horizontal scaling Sharding—detailing their roles, data flow, advantages, limitations, and operational considerations for building reliable, scalable databases.

Cluster ArchitectureDatabase ReplicationMongoDB
0 likes · 13 min read
MongoDB Cluster Architecture: Master‑Slave, Replica Set, and Sharding
Architects Research Society
Architects Research Society
Oct 25, 2023 · Operations

eBay’s Scalability Best Practices: Functional Partitioning, Horizontal Sharding, Avoiding Distributed Transactions, Asynchronous Decoupling, Caching, and Virtualization

The article outlines eBay’s key scalability best practices—including functional partitioning, horizontal sharding, eliminating distributed transactions, aggressive asynchronous decoupling, intelligent caching, and pervasive virtualization—to illustrate how large‑scale web systems can achieve linear or sub‑linear growth while maintaining availability and performance.

OperationsShardingasynchronous processing
0 likes · 14 min read
eBay’s Scalability Best Practices: Functional Partitioning, Horizontal Sharding, Avoiding Distributed Transactions, Asynchronous Decoupling, Caching, and Virtualization
Su San Talks Tech
Su San Talks Tech
Oct 19, 2023 · Databases

Mastering Database Sharding: Solving Read Amplification and Partition Strategies

This article explains database sharding concepts, compares vertical and horizontal partitioning, details ID‑range and modulo sharding methods, discusses the read‑amplification issue caused by non‑shard queries, and presents solutions such as auxiliary index tables, Elasticsearch integration, and TiDB as alternatives.

MySQLShardingTiDB
0 likes · 13 min read
Mastering Database Sharding: Solving Read Amplification and Partition Strategies
LouZai
LouZai
Oct 11, 2023 · Backend Development

Designing High-Concurrency Systems: Key Strategies and Best Practices

This article walks through essential techniques for building high‑concurrency systems, covering page static‑generation, CDN acceleration, caching layers, asynchronous processing with thread pools and MQ, sharding, connection pooling, read/write splitting, indexing, batch queries, clustering, load‑balancing, rate limiting, service degradation, failover, multi‑active deployment, stress testing, and monitoring.

Load BalancingMonitoringSharding
0 likes · 25 min read
Designing High-Concurrency Systems: Key Strategies and Best Practices
Selected Java Interview Questions
Selected Java Interview Questions
Sep 26, 2023 · Databases

Practical Guide to Monthly Horizontal Partitioning in MySQL Using Stored Procedures and SpringBoot

This article presents a hands‑on walkthrough of implementing in‑database horizontal partitioning by month for a high‑volume billing table, covering the business background, design decisions, stored‑procedure table creation, dynamic table selection in MyBatis, and query strategies such as serial‑number lookup and date‑range retrieval.

ShardingSpringBootStored Procedure
0 likes · 31 min read
Practical Guide to Monthly Horizontal Partitioning in MySQL Using Stored Procedures and SpringBoot
Senior Tony
Senior Tony
Sep 12, 2023 · Backend Development

What Really Powers High‑Concurrency Systems? Practical Solutions Explained

This article breaks down real‑world high‑concurrency strategies—horizontal scaling, caching, Elasticsearch, sharding, message‑queue smoothing, and cellization—explaining when each applies, their trade‑offs, and practical tips for building scalable, reliable backend services.

Message QueueScalingSharding
0 likes · 9 min read
What Really Powers High‑Concurrency Systems? Practical Solutions Explained
dbaplus Community
dbaplus Community
Sep 11, 2023 · Databases

When and How to Shard Databases: A Practical Guide to Splitting Tables and Schemas

This article explains why relational databases hit performance bottlenecks at large scale, introduces vertical and horizontal sharding techniques, compares their pros and cons, discusses common challenges such as distributed transactions, joins, pagination and global key generation, and offers practical solutions and middleware options.

Horizontal ShardingShardingVertical Sharding
0 likes · 24 min read
When and How to Shard Databases: A Practical Guide to Splitting Tables and Schemas
Shepherd Advanced Notes
Shepherd Advanced Notes
Aug 30, 2023 · Databases

Achieving Efficient Real‑Time Search for Massive Data in Spring Boot Applications

The article analyzes why massive tables become a bottleneck in Spring Boot systems, outlines the drawbacks of sharding, and presents a layered solution—data archiving, read‑write separation with caching, heterogeneous source synchronization via Elasticsearch and Canal, and selective sharding—to enable high‑performance real‑time search.

CanalData ArchivingElasticsearch
0 likes · 13 min read
Achieving Efficient Real‑Time Search for Massive Data in Spring Boot Applications
ITPUB
ITPUB
Aug 9, 2023 · Operations

Why Is My Elasticsearch Cluster Using 15 GB Heap? A Deep Dive into Memory Bottlenecks

The article examines a 7‑node Elasticsearch cluster with 500 million documents, uncovering excessive heap usage, high OS memory pressure, numerous deleted documents, large translog, low query‑cache hit rate, and an over‑sharded design, then offers concrete tuning and redesign recommendations to restore performance.

Cluster OperationsElasticsearchMemory Optimization
0 likes · 16 min read
Why Is My Elasticsearch Cluster Using 15 GB Heap? A Deep Dive into Memory Bottlenecks
ITPUB
ITPUB
Aug 7, 2023 · Databases

What Exactly Is a Distributed Database? Definitions, Features, and Architecture Explained

This article defines distributed databases, examines their external traits such as write‑heavy, low‑latency, massive concurrency, massive storage and high reliability, explores internal architectures like client‑side sharding, proxy middleware and unit‑based designs, compares them with Amazon Aurora, and summarizes key takeaways.

Distributed DatabasesOLTPSharding
0 likes · 19 min read
What Exactly Is a Distributed Database? Definitions, Features, and Architecture Explained
dbaplus Community
dbaplus Community
Jul 25, 2023 · Databases

How to Seamlessly Migrate a Live System to Sharding with Dual‑Write and Diff

This article explains how Qunar’s ticket ancillary service upgraded from a single-database architecture to a sharded one without downtime, detailing dual-write, transaction handling, mapping-key routing, diff verification, the challenges encountered, and a component-based solution that makes future migrations reusable.

Shardingdistributed transactiondual-write
0 likes · 35 min read
How to Seamlessly Migrate a Live System to Sharding with Dual‑Write and Diff
Programmer DD
Programmer DD
Jul 15, 2023 · Operations

Why Is My Elasticsearch Cluster Using 99% Memory? Sharding, Translog & JVM Insights

This article analyzes a 7‑node Elasticsearch cluster with 500 million documents, revealing excessive shard count, high heap and OS memory usage, large translog, low query‑cache hit rate, and heavy I/O, and offers concrete recommendations on mapping, sharding, JVM tuning, and resource management to restore performance.

Cluster MonitoringElasticsearchJVM
0 likes · 17 min read
Why Is My Elasticsearch Cluster Using 99% Memory? Sharding, Translog & JVM Insights
Architect
Architect
Jul 14, 2023 · Databases

From Single‑Node to Scalable Redis Cluster: A Step‑by‑Step Architecture Guide

This article walks through Redis's evolution from a simple single‑instance cache to a highly available, high‑performance cluster, explaining persistence mechanisms (RDB, AOF, hybrid), master‑slave replication, Sentinel automatic failover, and sharding strategies with concrete examples and trade‑offs.

ClusteringDatabase ArchitectureHigh Availability
0 likes · 20 min read
From Single‑Node to Scalable Redis Cluster: A Step‑by‑Step Architecture Guide
Su San Talks Tech
Su San Talks Tech
Jul 6, 2023 · Databases

Master ShardingSphere: Quick Sharding with Spring Boot and YML

This tutorial walks through the fundamentals and architecture of ShardingSphere, compares it with other sharding tools, and demonstrates how to implement database sharding in a Spring Boot project using both YML configuration and pure Java code, including detailed code snippets and deployment tips.

ShardingShardingSphereSpring Boot
0 likes · 22 min read
Master ShardingSphere: Quick Sharding with Spring Boot and YML
ITPUB
ITPUB
Jun 23, 2023 · Databases

Seamless Sharding Migration: Dual‑Write, Mapping Keys, and Diff Validation

This article explains how to smoothly migrate a legacy single‑database system to a sharded architecture by using dual‑write, mapping‑key routing, custom transaction handling, and offline/real‑time diff checks, while detailing the underlying MyBatis integration and component‑based design.

MyBatisShardingdatabase partitioning
0 likes · 36 min read
Seamless Sharding Migration: Dual‑Write, Mapping Keys, and Diff Validation
Wukong Talks Architecture
Wukong Talks Architecture
Jun 20, 2023 · Databases

Evolution of JD Baitiao’s Data Architecture: From MySQL to Apache ShardingSphere

This article chronicles JD Baitiao’s journey from early MySQL and NoSQL solutions through DBRep to the adoption of Apache ShardingSphere, highlighting the technical motivations, decoupling strategies, performance comparisons, and the broader Database Plus vision for scalable, stable financial‑grade data architectures.

Big DataDatabasesJD Baitiao
0 likes · 14 min read
Evolution of JD Baitiao’s Data Architecture: From MySQL to Apache ShardingSphere
JD Cloud Developers
JD Cloud Developers
Jun 15, 2023 · Databases

Mastering Distributed Join Queries: MySQL Sharding-JDBC and Elasticsearch Strategies

This article explores the challenges of distributed join queries, detailing MySQL sharding‑jdbc join implementation, routing strategies, and code examples, then examines Elasticsearch‑SQL join capabilities, various join algorithms, and practical considerations for using nested types, offering insights for optimizing performance in distributed data environments.

ElasticsearchMySQLSharding
0 likes · 11 min read
Mastering Distributed Join Queries: MySQL Sharding-JDBC and Elasticsearch Strategies
Qunar Tech Salon
Qunar Tech Salon
May 30, 2023 · Databases

Smooth Migration to Sharding: Design, Implementation, and Componentization of Qunar's Database Sharding Solution

This article describes the challenges of migrating a long‑running monolithic database to a sharded architecture, presents a two‑stage smooth migration strategy using double‑write, diff verification, and mapping‑key techniques, and details the componentized design, core implementation, and configuration of a reusable sharding framework built on MyBatis and Spring.

Data synchronizationDatabase MigrationMyBatis
0 likes · 34 min read
Smooth Migration to Sharding: Design, Implementation, and Componentization of Qunar's Database Sharding Solution
Top Architect
Top Architect
May 29, 2023 · Databases

Large Table Splitting in MySQL: Challenges, Solutions, and Lessons Learned

Facing a 50‑million‑row financial transaction table, the team implemented horizontal sharding with sharding‑jdbc, designed migration and pagination strategies, tackled multi‑datasource transaction issues, and outlined a phased rollout, providing practical insights for large‑scale MySQL table splitting and backend system stability.

Data MigrationMySQLSharding
0 likes · 11 min read
Large Table Splitting in MySQL: Challenges, Solutions, and Lessons Learned
MaGe Linux Operations
MaGe Linux Operations
May 16, 2023 · Databases

How We Split a 50‑Million‑Row MySQL Table: Strategies and Lessons Learned

This article recounts how a finance team tackled a 50‑million‑row MySQL table by analyzing system bottlenecks, defining splitting goals, selecting sharding‑jdbc, addressing multi‑datasource transactions and pagination, designing data migration strategies, and executing a staged rollout to ensure stability and performance.

Database MigrationLarge TablesMySQL
0 likes · 12 min read
How We Split a 50‑Million‑Row MySQL Table: Strategies and Lessons Learned
Wukong Talks Architecture
Wukong Talks Architecture
May 16, 2023 · Databases

Understanding Sharding: 21 Common Concepts of Database Partitioning

This article introduces the fundamental concepts of database sharding, including data fragmentation, nodes, logical and physical tables, broadcast and single tables, sharding keys, strategies, algorithms, binding tables, SQL parsing, routing, rewriting, execution, result merging, distributed primary keys, data masking, distributed transactions, migration, and shadow databases, providing a comprehensive guide for designing and operating a sharded architecture.

Broadcast TableSQL RoutingSharding
0 likes · 19 min read
Understanding Sharding: 21 Common Concepts of Database Partitioning
Architect
Architect
May 13, 2023 · Databases

Splitting a Massive MySQL Financial Transaction Table: Challenges, Strategies, and Implementation

The article details how a finance team tackled a 50‑million‑row MySQL transaction table by analyzing the pre‑split problems, defining split goals, choosing sharding‑jdbc, addressing multi‑datasource transaction and pagination issues, designing a phased migration and rollout plan, and summarizing lessons learned.

Data MigrationMySQLSharding
0 likes · 13 min read
Splitting a Massive MySQL Financial Transaction Table: Challenges, Strategies, and Implementation
ITPUB
ITPUB
Apr 22, 2023 · Databases

Choosing the Right Sharding Algorithm: From Range to Consistent Hash

This article compares common database sharding strategies—range sharding, mapping tables, ID modulo, hash‑based sharding, and consistent hashing—explaining their implementation complexity, data distribution characteristics, scalability concerns, and practical trade‑offs with code examples.

Consistent HashHashingSharding
0 likes · 11 min read
Choosing the Right Sharding Algorithm: From Range to Consistent Hash
dbaplus Community
dbaplus Community
Apr 19, 2023 · Databases

How to Migrate Large Tables and Sync Data Seamlessly in Production

This article explains common business scenarios such as large‑table splitting, cross‑database migration, and data synchronization, then compares loss‑bearing and loss‑less migration strategies, and provides detailed step‑by‑step implementation guidance for smooth dual‑write migration and real‑world tooling.

Data MigrationDatabasesSharding
0 likes · 17 min read
How to Migrate Large Tables and Sync Data Seamlessly in Production
Efficient Ops
Efficient Ops
Apr 18, 2023 · Databases

Mastering MongoDB Clusters: Setup, Monitoring, Migration, and Optimization

This comprehensive guide explains MongoDB cluster architecture, component roles, common use cases, monitoring commands, essential maintenance operations, data migration steps, troubleshooting of typical production issues, and practical optimization recommendations for high‑performance deployments.

BackupClusterMongoDB
0 likes · 20 min read
Mastering MongoDB Clusters: Setup, Monitoring, Migration, and Optimization
IT Architects Alliance
IT Architects Alliance
Apr 14, 2023 · Databases

Comprehensive Guide to Database Horizontal Scaling, Smooth 2N Expansion, and Keepalived High‑Availability Configuration

This technical guide explains how to scale a sharded database horizontally by introducing five expansion schemes—including shutdown, write‑stop, log‑based, dual‑write, and smooth 2N approaches—covers MariaDB installation, master‑master replication setup, dynamic data‑source configuration with ShardingJDBC, and detailed Keepalived high‑availability configuration for seamless service continuity.

Database ScalingDynamic Data SourceHigh Availability
0 likes · 31 min read
Comprehensive Guide to Database Horizontal Scaling, Smooth 2N Expansion, and Keepalived High‑Availability Configuration
Top Architect
Top Architect
Mar 14, 2023 · Databases

Handling Large MySQL Tables: Partitioning, Sharding, and Archiving Strategies

This article explains how to assess MySQL table size, identify performance issues caused by massive data volumes, and presents three practical solutions—table partitioning, database sharding, and hot‑cold data archiving—along with detailed SQL examples and considerations for implementation.

Data ArchivingMySQLSharding
0 likes · 16 min read
Handling Large MySQL Tables: Partitioning, Sharding, and Archiving Strategies
Su San Talks Tech
Su San Talks Tech
Mar 14, 2023 · Databases

How to Solve Read‑Amplification in MySQL Sharding: Strategies & Tips

This article explains the concepts of database sharding, compares vertical and horizontal partitioning, discusses ID‑range and modulo sharding methods, analyzes the read‑amplification problem caused by non‑sharding keys, and presents practical solutions such as auxiliary index tables, Elasticsearch integration, and TiDB adoption.

Database ScalingElasticsearchHorizontal Partitioning
0 likes · 14 min read
How to Solve Read‑Amplification in MySQL Sharding: Strategies & Tips
Top Architect
Top Architect
Mar 4, 2023 · Databases

How to Handle Large MySQL Tables: Partitioning, Sharding, and Archiving Strategies

This article explains why large MySQL tables cause slow queries, analyzes table size, disk usage, and instance capacity, and presents three practical solutions—table partitioning, database sharding (horizontal/vertical), and hot‑cold data archiving—helping developers choose the appropriate strategy for their workloads.

Data ArchivingMySQLSharding
0 likes · 18 min read
How to Handle Large MySQL Tables: Partitioning, Sharding, and Archiving Strategies
MaGe Linux Operations
MaGe Linux Operations
Feb 22, 2023 · Databases

Mastering MySQL Sharding: When and How to Use Database Partitioning

This article explains the concepts, strategies, and practical scenarios for MySQL database partitioning, covering both table splitting (vertical and horizontal) and database splitting (vertical and horizontal), with examples, advantages, and when to apply each technique to handle large-scale data workloads.

Horizontal PartitioningShardingVertical Partitioning
0 likes · 9 min read
Mastering MySQL Sharding: When and How to Use Database Partitioning
ITPUB
ITPUB
Feb 18, 2023 · Backend Development

Mastering High‑Performance Backend: Lock‑Free, Zero‑Copy, Serialization, and More

This comprehensive guide explores essential backend techniques—including lock‑free programming, zero‑copy I/O, efficient serialization, pooling, concurrency, async processing, caching strategies, sharding, storage optimizations, and queue mechanisms—to build high‑performance, scalable services while highlighting practical code examples and real‑world trade‑offs.

Lock-FreeQueueSerialization
0 likes · 51 min read
Mastering High‑Performance Backend: Lock‑Free, Zero‑Copy, Serialization, and More
Tencent Cloud Developer
Tencent Cloud Developer
Feb 15, 2023 · Databases

How to Build a Stable, High‑Performance Redis Cluster: Persistence, Replication, Sentinel and Sharding

This guide walks developers through transforming a single‑node Redis cache into a stable, high‑performance cluster by explaining data persistence options (RDB, AOF, hybrid), master‑slave replication, Sentinel‑driven automatic failover, and both client‑side and proxy‑based sharding for horizontal scalability.

ClusterPersistenceRedis
0 likes · 17 min read
How to Build a Stable, High‑Performance Redis Cluster: Persistence, Replication, Sentinel and Sharding
Shepherd Advanced Notes
Shepherd Advanced Notes
Feb 13, 2023 · Backend Development

Elasticsearch Advanced Guide: Core Concepts and Implementation Principles

This article provides a comprehensive technical overview of Elasticsearch, covering core concepts such as indexes, types, documents, fields, mappings, shards and replicas, the cluster architecture, routing, write and read workflows, segment handling, translog, refresh and flush mechanisms, as well as practical optimization recommendations for hardware, shard strategy, routing, write performance and memory configuration.

Cluster ArchitectureElasticsearchIndexing
0 likes · 36 min read
Elasticsearch Advanced Guide: Core Concepts and Implementation Principles
Efficient Ops
Efficient Ops
Feb 12, 2023 · Databases

Why Single‑Node MySQL Is No Longer Enough: Embrace Distributed Databases

The article explains the limitations of standalone MySQL for large‑scale workloads, explores sharding, read/write separation, and middleware approaches, and argues that modern distributed databases and cloud‑native solutions are the sustainable path forward for data‑intensive applications.

Cloud databasesDatabase ScalingDistributed Databases
0 likes · 6 min read
Why Single‑Node MySQL Is No Longer Enough: Embrace Distributed Databases
21CTO
21CTO
Feb 11, 2023 · Big Data

Mastering Elasticsearch: Core Concepts, Architecture, and Performance Tips

This comprehensive guide explains what Elasticsearch does, its underlying Lucene technology, core concepts such as clusters, shards, replicas, mapping, indexing and storage mechanisms, and provides practical performance‑tuning advice for building and operating a robust distributed search engine.

ClusterElasticsearchSharding
0 likes · 35 min read
Mastering Elasticsearch: Core Concepts, Architecture, and Performance Tips
dbaplus Community
dbaplus Community
Jan 29, 2023 · Databases

Why Single‑Node MySQL Is Dying and Distributed Databases Are the Future

The article explains how single‑instance MySQL struggles with huge tables, resource‑starved databases, and high read traffic, and argues that sharding middleware is being replaced by modern distributed databases such as TiDB, OceanBase, Aurora, and PolarDB as the sustainable scaling solution.

Database ArchitectureMySQLSharding
0 likes · 9 min read
Why Single‑Node MySQL Is Dying and Distributed Databases Are the Future
ITPUB
ITPUB
Jan 29, 2023 · Databases

How to Query Sharded Orders Efficiently: Redundant, Index, and Gene Strategies

When order_id is not the sharding key, querying a sharded order table can require scanning every shard, but three techniques—data redundancy, an index table, and embedding shard information into the order_id (gene method)—enable fast, single‑shard lookups.

Shardinggene methodindex table
0 likes · 10 min read
How to Query Sharded Orders Efficiently: Redundant, Index, and Gene Strategies
Architect's Guide
Architect's Guide
Jan 26, 2023 · Databases

Primary Key and Index Design for Distributed Databases

The article explains why auto‑increment primary keys are unsuitable for sharded databases, recommends globally unique ordered identifiers, and details index and global‑table strategies—including redundant tables, index tables, and embedding shard information in keys—to achieve efficient single‑shard queries.

Distributed DatabasesSQLSharding
0 likes · 10 min read
Primary Key and Index Design for Distributed Databases
Top Architect
Top Architect
Jan 12, 2023 · Databases

MySQL Master‑Slave Replication, Binlog/Redo Log Basics, and Sharding‑JDBC Read‑Write Splitting & Sharding Implementation

This article provides a comprehensive tutorial on configuring MySQL master‑slave replication with Docker, explains the roles of binlog and redo log, describes two‑phase commit, and demonstrates how to use Sharding‑JDBC (ShardingSphere) for read‑write splitting and horizontal sharding in a Spring Boot application, complete with configuration files and code examples.

DatabaseMySQLSharding
0 likes · 24 min read
MySQL Master‑Slave Replication, Binlog/Redo Log Basics, and Sharding‑JDBC Read‑Write Splitting & Sharding Implementation
ITPUB
ITPUB
Dec 31, 2022 · Databases

15 Must‑Know Interview Questions on Database Sharding and Partitioning

This article explains why and when to split databases and tables, how to choose sharding keys, various sharding strategies such as range, hash and consistent hash, handling cross‑node joins, pagination, distributed IDs, middleware choices, and step‑by‑step zero‑downtime migration techniques.

MySQLShardingscalability
0 likes · 17 min read
15 Must‑Know Interview Questions on Database Sharding and Partitioning
Java Backend Technology
Java Backend Technology
Dec 29, 2022 · Databases

Why MySQL Must Evolve: From Single‑Node Limits to Distributed Databases

The article explains how traditional single‑node MySQL faces scalability challenges such as massive tables, resource exhaustion, and read‑heavy workloads, and argues that adopting sharding, middleware, or modern distributed databases like TiDB, OceanBase, Aurora, and PolarDB is the inevitable path forward for robust data management.

Cloud DatabaseDatabase ScalingMySQL
0 likes · 6 min read
Why MySQL Must Evolve: From Single‑Node Limits to Distributed Databases
Architect
Architect
Dec 28, 2022 · Backend Development

Design and Implementation of Pickup Code Generation and Write‑off Logic for SaaS E‑commerce Platforms

This article explains the end‑to‑end design of a pickup code system—including simple single‑table implementation, sharding‑aware multi‑database strategies, code generation algorithms, concurrency handling with distributed locks, and practical solutions to common pitfalls—targeted at SaaS e‑commerce back‑ends.

SaaSShardingorder management
0 likes · 12 min read
Design and Implementation of Pickup Code Generation and Write‑off Logic for SaaS E‑commerce Platforms
Top Architect
Top Architect
Dec 24, 2022 · Databases

Elasticsearch Architecture: Inverted Index, Sharding, and Data Operations

This article explains the core concepts of Elasticsearch, including how its inverted index works, the structure of term dictionaries and posting lists, shard and replica configuration, cluster node roles, the detailed write, refresh, flush, and merge processes, as well as how search queries are executed across distributed shards.

Distributed ArchitectureShardinginverted index
0 likes · 9 min read
Elasticsearch Architecture: Inverted Index, Sharding, and Data Operations
Architect
Architect
Dec 23, 2022 · Databases

Understanding Elasticsearch Inverted Index, Sharding, and Data Operations

This article explains the core concepts of Elasticsearch, including the structure and purpose of inverted indexes, how shards and replicas are organized in a cluster, and the detailed workflows for writing, reading, searching, and deleting documents within a distributed environment.

Data WriteShardingdistributed-architecture
0 likes · 8 min read
Understanding Elasticsearch Inverted Index, Sharding, and Data Operations
ITPUB
ITPUB
Dec 8, 2022 · Databases

How to Efficiently Query Sharded Order Tables: Redundant, Index, and Gene Methods

When order_id is not the sharding key, querying a sharded order table can require scanning every shard, but three design patterns—data redundancy, an index table, and gene‑based sharding—enable fast single‑shard lookups while balancing storage and complexity.

Database DesignShardinggene sharding
0 likes · 9 min read
How to Efficiently Query Sharded Order Tables: Redundant, Index, and Gene Methods
JD Tech
JD Tech
Dec 7, 2022 · Databases

Practical Guide to Large‑Scale Data Migration Using Sharding‑Proxy

This article presents a step‑by‑step practice of migrating massive billing data to 32 sharded databases with Sharding‑Proxy, covering background, objectives, four migration solutions, detailed proxy installation and configuration, debugging, migration workflow, data validation, common issues and their resolutions.

ConfigurationData MigrationDatabase
0 likes · 14 min read
Practical Guide to Large‑Scale Data Migration Using Sharding‑Proxy
ITPUB
ITPUB
Nov 29, 2022 · Databases

When MySQL Hits Its Limits: Scaling with Horizontal and Vertical Sharding

This article explains MySQL's theoretical row limits, why performance degrades long before reaching them, and how horizontal and vertical database sharding can distribute massive datasets across multiple instances to improve load handling and enable seamless scaling.

Horizontal PartitioningMySQLSharding
0 likes · 7 min read
When MySQL Hits Its Limits: Scaling with Horizontal and Vertical Sharding
Programmer DD
Programmer DD
Nov 21, 2022 · Databases

How to Achieve Seamless Horizontal Scaling with 2N Expansion and Keepalived

This guide explains how to scale a sharded database horizontally by adding new nodes, handling data migration with stop‑service, stop‑write, log‑based, dual‑write, and smooth 2N strategies, and implementing high‑availability using MariaDB double‑master replication and Keepalived, complete with configuration examples and code snippets.

Data MigrationHigh AvailabilityKeepalived
0 likes · 35 min read
How to Achieve Seamless Horizontal Scaling with 2N Expansion and Keepalived
ITPUB
ITPUB
Nov 19, 2022 · Databases

Choosing Between OceanBase and TiDB: Architecture, Performance, and Trade‑offs

This article compares the architectures of OceanBase and TiDB, explaining how their sharding and compute‑storage separation designs affect high availability, latency, and workload suitability, and offers guidance on selecting the right distributed database for complex enterprise applications.

Compute-Storage SeparationOceanBaseSharding
0 likes · 12 min read
Choosing Between OceanBase and TiDB: Architecture, Performance, and Trade‑offs
Architect's Guide
Architect's Guide
Nov 11, 2022 · Databases

Database Sharding: Table Partitioning, Horizontal and Vertical Splitting, and Implementation with Sharding-JDBC and Mycat

This article explains why high‑concurrency big‑data systems need database sharding, describes table‑level partitioning methods (modulus and range), shows how to implement them with MySQL and Java code, compares vertical and horizontal database splitting, and evaluates middleware solutions such as Sharding‑JDBC and Mycat.

MySQLShardingSpring
0 likes · 11 min read
Database Sharding: Table Partitioning, Horizontal and Vertical Splitting, and Implementation with Sharding-JDBC and Mycat
Top Architect
Top Architect
Nov 10, 2022 · Databases

Database Sharding and Partitioning Strategy for High‑Volume Order Systems

This article explains how to handle billions of order records by classifying data into hot and cold tiers, using MySQL sharding with database‑and‑table partitioning, storing cold data in Elasticsearch and Hive, and outlines combined routing formulas for scalable backend architecture.

MySQLShardingdatabase partitioning
0 likes · 8 min read
Database Sharding and Partitioning Strategy for High‑Volume Order Systems
Architects Research Society
Architects Research Society
Nov 4, 2022 · Fundamentals

eBay Scalability Best Practices: Functional Partitioning, Horizontal Sharding, Async Decoupling, and More

This article outlines eBay's key scalability best practices—including functional decomposition, horizontal sharding, avoiding distributed transactions, asynchronous decoupling, virtualization, and intelligent caching—to demonstrate how large‑scale web systems can achieve linear resource growth and high availability.

AsynchronousDatabaseOperations
0 likes · 14 min read
eBay Scalability Best Practices: Functional Partitioning, Horizontal Sharding, Async Decoupling, and More