The article analyzes why traditional LIMIT/OFFSET pagination fails when data is split across multiple databases, presents a global query approach with its trade‑offs, and proposes an optimized "no‑skip" method plus practical tips using ShardingSphere and Elasticsearch.
The article analyzes why traditional offset‑based pagination breaks after data is split across multiple databases, explains the global query method with its pros and cons, and introduces the no‑skip optimization that reduces data transfer and improves deep‑page performance.
In high‑concurrency internet architectures, a MySQL master‑slave setup with read‑write splitting is the baseline for high availability, but static routing suffers from node failures and lag; this article explains how ShardingSphere provides health checks, auto‑failover, load‑balancing, and degradation to achieve resilient read‑write separation.
This article explains why single‑database, single‑table architectures hit performance limits at millions of rows, introduces ShardingSphere as the preferred zero‑deployment sharding solution for SpringBoot, and walks through vertical and horizontal splitting, configuration, custom algorithms, read/write splitting, and distributed transactions with practical code examples.
The article explains vertical and horizontal sharding concepts, compares their advantages and drawbacks, evaluates popular sharding middleware such as ShardingSphere, details distributed ID generation with Snowflake, offers cross‑database query solutions, and provides practical guidance on when and how to apply sharding in MySQL.
This tutorial walks you through configuring Spring Boot 3.5.5 with ShardingSphere‑JDBC 5.5.2 to achieve database sharding, read‑write splitting, connection‑pool optimization using Druid, and demonstrates the complete code for entities, mappers, services, and controllers, plus verification steps.
This article examines the challenges of a growing order system, compares sharding and distributed database options, evaluates unique ID generation methods, outlines a migration plan using ShardingSphere‑JDBC and Seata, and provides risk mitigation and best‑practice recommendations for large‑scale backend databases.
The article analyzes a JD interview question on scaling a MySQL order table from 100 million to billions of rows, detailing four practical expansion strategies—stop‑service migration, double‑write migration, ShardingSphere middleware, and dual‑VIP routing—each with step‑by‑step procedures, code samples, risk assessments, and validation methods to enable near‑instant capacity growth without service interruption.
This article walks through building a Spring Boot project with ShardingSphere‑JDBC, covering Maven dependencies, YAML configuration, database setup, sharding rules, distributed ID generation, binding tables, and testing, providing a complete, hands‑on guide to implementing MySQL horizontal partitioning in a real‑world application.
An unexpected data loss issue in a ShardingSphere‑JDBC setup was traced to Druid’s poolPreparedStatements cache, which reuses PreparedStatement objects across different shards, bypassing dynamic routing; disabling the cache or switching to HikariCP restores correct sharding behavior with minimal performance impact.
This article deeply analyzes ShardingSphere-JDBC 5.3.0 source code, explaining its JDBC‑compatible sharding design, modular architecture, adapter pattern implementation, SQL parsing and routing, distributed transaction support, elastic scaling, performance optimizations, and extensibility via SPI.
This comprehensive tutorial explains database sharding concepts, types, strategies, and implementation in Spring Boot using ShardingSphere, covering configuration, entity and repository code, service and controller layers, integration with pagination, Swagger, ActiveMQ, security, batch processing, FreeMarker, WebSockets, AOP, performance testing, FAQs, real‑world cases, and future trends.
This article walks through the end‑to‑end design, configuration, and implementation of splitting massive loan and repayment tables into 50 sharded MySQL tables using ShardingSphere, Spring Boot, and a custom suffix algorithm, while covering data migration, DBA coordination, dynamic switches, and scheduled consistency checks.
This guide shows how to combine MyBatis-Plus dynamic data source routing with ShardingSphere-JDBC for horizontal sharding and read‑write splitting, including Maven dependencies, YAML configuration, Spring bean setup, and a custom load‑balancing algorithm to keep @DS annotations functional.
This article walks through the complete process of designing, configuring, and coding a sharding solution for loan and repayment tables—including database basics, historical data migration, backend query refactoring, a three‑write strategy, scheduled consistency checks, and full SpringBoot/ShardingSphere implementation details.
This guide demonstrates how to set up Docker‑based MySQL instances, configure ShardingSphere‑Proxy for database sharding and read‑write splitting, and integrate ShardingSphere‑JDBC with SpringBoot 2.x using YAML configurations and Maven dependencies, providing complete code snippets and default connection details.
This article introduces ShardingSphere's three components—Sharding-JDBC, Sharding-Proxy, and the upcoming Sharding-Sidecar—explains why sharding is needed for relational databases, demonstrates horizontal data sharding concepts, and provides a complete Spring Boot example with configuration, code, and asynchronous batch insertion.
This comprehensive guide walks through ShardingSphere's core sharding strategies and the twelve built‑in sharding algorithms, showing how to configure each with YAML, illustrating their behavior with code examples, and highlighting practical tips for choosing the right approach in real‑world database partitioning scenarios.
This article describes how a senior architect designed, configured, and implemented a 50‑table sharding solution for loan and repayment request data using ShardingSphere, Spring Boot 3, MySQL, and custom synchronization scripts, while also addressing historical data migration, backend query changes, and operational safeguards.
This article describes how a senior architect designed and implemented a sharding solution for loan and repayment tables, covering database design, table naming, sharding algorithms, historical data migration, three‑write synchronization, dynamic switches, and the necessary Spring Boot, MyBatis‑Plus, and ShardingSphere configurations with code examples.
The article outlines the growing difficulties of managing diverse databases in cloud-native environments, introduces Apache ShardingSphere as a comprehensive open‑source solution with three core capabilities—connectivity, enhancement, and pluggability—and guides readers through a three‑step learning path from fundamentals to deployment and testing.
The article describes a year‑long development experience where a senior architect designs, plans, and implements a sharding solution for loan and repayment tables using ShardingSphere, Spring Boot 3, MySQL, and custom synchronization code to migrate billions of rows while ensuring data consistency and operational continuity.
This article details the design, planning, and step‑by‑step implementation of a database table‑partitioning solution for loan and repayment request tables, covering schema decisions, sharding strategy, historical data migration, dynamic synchronization, code examples with ShardingSphere, SpringBoot, and operational considerations for seamless migration.
The article describes how a one‑year‑old loan/repayment service was refactored by introducing a 50‑table sharding scheme with ShardingSphere, detailing the design decisions, historical data migration, code changes, configuration files, pitfalls, and runtime synchronization strategies to achieve efficient query performance on billions of rows.
This article details the end‑to‑end design, configuration, and code implementation of a 50‑table sharding solution for cash repayment and loan application data, covering database schema planning, historical data migration, backend query adaptation, dynamic switches, scheduled consistency checks, and practical pitfalls encountered with Spring Boot 3 and ShardingSphere.
This article explains database sharding concepts, reasons for using it, various sharding strategies, and demonstrates implementation with ShardingSphere, including configuration and Java code examples, while also offering additional resources for advanced architecture and interview preparation.
This article introduces ShardingSphere, an open‑source distributed database middleware, explains its core features such as horizontal sharding, read‑write separation, and distributed transactions, and provides a step‑by‑step configuration and Java usage example.
This article explains the difference between database sharding and table partitioning, illustrates scaling challenges with a fast‑growing startup scenario, and compares popular sharding middleware such as Cobar, TDDL, Atlas, ShardingSphere, and Mycat, offering practical recommendations for different company sizes.
This guide explains the two integration mechanisms ShardingSphere provides for Spring—namespace‑based XML schema extension and custom starter for Spring Boot—detailing the required classes, configuration files, bean definitions, and annotations to enable sharding, read‑write splitting, and encryption in a Java backend.
This article explores why distributed primary keys are essential for sharding, compares built‑in ShardingSphere ID generation strategies such as UUID, NanoID, Snowflake, CosId and custom algorithms, details their configuration, performance trade‑offs, and provides code examples for implementing and registering custom key generators.
This article explains the fundamentals of database sharding and partitioning, why they are needed for massive data volumes, various vertical and horizontal splitting strategies, routing algorithms, client vs. proxy deployment models, and the practical challenges and solutions when using ShardingSphere 5.x.
This article provides a comprehensive, step‑by‑step tutorial on using Apache ShardingSphere and Sharding‑JDBC to implement horizontal, vertical, and broadcast sharding, covering core concepts, sharding strategies, configuration examples, and practical code snippets for real‑world database partitioning.
This article introduces database middleware, explains why it’s needed for scaling and performance, and provides a detailed overview of six popular middleware solutions—including ShardingSphere, MyCat, MySQL‑Proxy, DRDS, Atlas, and Zebra—highlighting their core features and use cases.
This article explains what database middleware is, why it’s needed for scaling large data workloads, and provides detailed overviews of six popular middleware solutions—including MyCat, MySQL‑Proxy, DRDS, Atlas, Zebra, and ShardingSphere—highlighting their key features and use cases.
This article compiles a set of practical SpringBoot techniques, including Flyway-based SQL version control, JetCache declarative caching, Netty WebSocket service customization, jasypt configuration encryption, ShardingSphere data masking, Jackson response desensitization, read‑write splitting, idempotent request handling, MockMvc testing, and Prometheus‑Grafana monitoring.
This article examines how JD Logistics tackled MySQL connection limits by redesigning its sharding strategy using ShardingSphere's Sharding-JDBC, detailing the background, concepts of database and table sharding, analysis of bottlenecks, and step‑by‑step refactoring to reduce connections and improve scalability.
This tutorial walks through the fundamentals and architecture of ShardingSphere, compares it with other sharding tools, and demonstrates how to implement database sharding in a Spring Boot project using both YML configuration and pure Java code, including detailed code snippets and deployment tips.
This article chronicles JD Baitiao’s journey from early MySQL and NoSQL solutions through DBRep to the adoption of Apache ShardingSphere, highlighting the technical motivations, decoupling strategies, performance comparisons, and the broader Database Plus vision for scalable, stable financial‑grade data architectures.
This guide walks through setting up a Spring Boot 2.6.14 project with ShardingSphere 5.3.0 encryption, covering Maven dependencies, YAML configurations for data sources and encryptors, defining the users table schema, implementing CRUD operations with MyBatis‑Plus, and troubleshooting connection‑pool settings.
This article provides a comprehensive tutorial on configuring MySQL master‑slave replication with Docker, explains the roles of binlog and redo log, describes two‑phase commit, and demonstrates how to use Sharding‑JDBC (ShardingSphere) for read‑write splitting and horizontal sharding in a Spring Boot application, complete with configuration files and code examples.
iQIYI adopts SphereEx’s Database Mesh 2.0 by extending ShardingSphere‑JDBC and integrating the cloud‑native Pisanix proxy, creating a unified, encrypted configuration center that enables dynamic sharding, load‑balancing, read‑write separation, hot‑updates and observability, dramatically simplifying database governance and cloud migration.
This article explains the fundamentals of database sharding, why and when to apply it, various vertical and horizontal splitting strategies, routing algorithms, common pitfalls such as pagination and distributed transactions, and compares client‑side and proxy‑based ShardingSphere deployment patterns.
This article introduces Sharding-JDBC and ShardingSphere, explains their history, core architecture, vertical and horizontal sharding concepts, sharding rule configuration, SQL parsing and rewriting, routing and result merging, and provides a quick‑start guide with Maven dependencies and Spring Boot configuration for Java applications.
The article explains why microservice governance is essential, introduces the OpenSergo project and its collaboration with ShardingSphere to create a unified, language‑agnostic standard for database governance, and provides concrete abstractions and YAML examples for implementing traffic, fault‑tolerance, and security policies.
The article details a NullPointerException that occurs when ShardingSphere starts, explains how null values in TiDB‑generated view metadata trigger the error, describes the step‑by‑step investigation and reproduction using TiUP, and discusses several possible fixes and a call for community contributions.
The article details the design and implementation of a pickup‑code system for SaaS e‑commerce platforms, covering simple single‑table approaches and a globally unique 8‑digit code scheme that embeds database and table identifiers, with strategies for sharding, conflict resolution, dynamic routing, and extensible tenant‑specific configuration.
This tutorial walks through the complete process of configuring MySQL master‑slave replication, preparing Docker containers, configuring MyBatisPlus and ShardingSphereJDBC for read‑write splitting, building a Spring Boot project with the necessary dependencies, generating code, and testing the setup with sample controller endpoints.
This tutorial demonstrates how to set up MySQL master‑slave replication using Docker, configure MyBatisPlus and ShardingSphereJDBC for read/write splitting, and integrate the setup into a Spring Boot application with Druid, code generation, and REST endpoints for testing.
This tutorial explains how to configure MySQL master‑slave replication using Docker, set up read‑write splitting with ShardingSphereJDBC, integrate MyBatis‑Plus, and build a Spring Boot application that demonstrates write operations on the master and read operations on the slave.
This tutorial walks through setting up MySQL master‑slave replication using Docker, configuring ShardingSphereJDBC for read‑write splitting, and integrating it into a Spring Boot project with MyBatis‑Plus, covering environment preparation, core theory, detailed configuration, and testing procedures.
This article explains the concept, benefits, and step‑by‑step configuration of non‑intrusive read‑write splitting for MySQL databases using MSE and OpenSergo, including practical examples, YAML rules, and SQL insight for improved stability and performance.
To support the rapidly growing DeWu App, the team created Rainbow Bridge—a ShardingSphere‑based middleware that unifies horizontal sharding, read‑write separation, shadow‑DB routing, rate limiting, and active‑active traffic correction, while adding a graphical console, stability enhancements, fine‑grained observability, hybrid JDBC/Proxy switching, and ongoing open‑source contributions.
This article provides a step‑by‑step tutorial on configuring MySQL master‑slave replication with Docker, explains redo log and binlog fundamentals, demonstrates two‑phase commit, and shows how to implement read‑write splitting and sharding using ShardingSphere‑Jdbc and Sharding‑Jdbc with detailed code examples.
This tutorial explains why read/write splitting is useful, shows step‑by‑step how to build a MySQL master‑slave cluster with Docker, configure the master and slave instances, and then use Sharding‑JDBC to transparently route writes to the master and reads to the slave.
This article chronicles the architectural evolution of JD Baitiao’s backend—from early MySQL monoliths through Solr‑HBase, MongoDB, and DBRep—to the adoption of Apache ShardingSphere, highlighting the motivations, technical trade‑offs, decoupling strategies, and performance outcomes for a high‑throughput financial service.
This article provides a comprehensive tutorial on using ShardingSphere‑JDBC, configuring MySQL master‑slave replication with Docker, explaining binlog and redo‑log mechanisms, demonstrating two‑phase commit, and detailing Sharding‑JDBC read/write splitting and sharding configurations with code examples.
This article details how ZhaiZhai tackled severe order database performance and capacity issues by adopting ShardingSphere for sharding, describing the challenges, mitigation steps, shard key design, migration process, and the significant performance improvements achieved.
The article walks through the challenges of growing data volumes, compares master‑slave, vertical, horizontal, and sharding‑plus‑partitioning architectures, explains their pros and cons, and provides concrete guidance on selecting and implementing a suitable database solution, including middleware options and real‑world case studies.
This article explains how to use Apache ShardingSphere's data desensitization module with Spring and Spring Boot to transparently encrypt sensitive fields such as ID numbers and bank cards, covering pain points, configuration steps, code examples, and datasource integration for compliance‑driven applications.
This article provides a step‑by‑step tutorial on configuring MySQL master‑slave replication with Docker, explains redo log and binlog mechanisms, demonstrates two‑phase commit, and shows how to use ShardingSphere‑Jdbc for read/write splitting and sharding with detailed Spring Boot examples and code snippets.
This article chronicles JD Baitiao's data‑architecture evolution from Solr + HBase to MongoDB and finally to Apache ShardingSphere, highlighting the challenges of massive data growth, the need for decoupling, and the performance, scalability, and operational benefits achieved by adopting ShardingSphere.
This article provides a comprehensive guide on using ShardingSphere-JDBC for read‑write splitting, configuring MySQL master‑slave replication with Docker, explaining redo log and binlog mechanisms, demonstrating two‑phase commit, and showing how to implement database sharding and distributed primary keys in a Spring Boot application.
This article introduces Sharding-JDBC (now ShardingSphere), explains core sharding concepts such as shards, data nodes, logical and physical tables, sharding keys, algorithms and strategies, shows how it extends JDBC, and provides a step‑by‑step Spring Boot + MyBatis‑Plus example for building a sharded database application.
This article explains why encrypting sensitive customer data in databases is essential, outlines common pain points, and demonstrates how Apache ShardingSphere’s data‑desensitization module can be quickly configured in Spring (XML and Boot) to provide transparent AES encryption and decryption without altering business code.
The article explains how to transition from client‑side sharding with Sharding‑JDBC to proxy‑side sharding using Rainbow Bridge DAL by adding new, switchable data sources, employing AbstractRoutingDataSource and configuration flags for gradual migration, while addressing read‑write separation, unsupported SQL functions, and ensuring backward compatibility.
This article provides a detailed walkthrough of ShardingSphere‑Jdbc read/write splitting, MySQL master‑slave replication setup with Docker, the differences between redo log and binlog, two‑phase commit, and practical sharding configurations using inline strategies and distributed primary keys in a Spring Boot project.
This article explains how to use Apache ShardingSphere's encryption module to transparently store and query sensitive customer data such as ID numbers and bank cards by configuring encryption rules in Spring or Spring Boot, eliminating manual SQL encryption and simplifying compliance for legacy systems.
This tutorial explains why encrypting sensitive fields like ID numbers and bank cards is required, outlines common pain points, and shows how to configure ShardingSphere's data‑desensitization module using Spring namespace or Spring Boot with full code examples and configuration files.
This article introduces ShardingSphere‑Jdbc as a lightweight Java JDBC framework, details step‑by‑step Docker‑based MySQL master‑slave replication configuration, reviews redo log and binlog mechanisms, explains two‑phase commit, and demonstrates read‑write splitting and sharding strategies with Spring Boot examples.
This article explains why sensitive customer data must be encrypted at rest, outlines the drawbacks of manual SQL encryption and retrofitting existing systems, and provides step‑by‑step Spring and Spring Boot configurations using ShardingSphere's encryption module with full code examples.
This article explains how to use Spring Boot to quickly integrate ShardingSphere for database sharding, covering the background of massive data storage challenges, the advantages of ShardingSphere, step‑by‑step project setup, configuration of data sources and sharding strategies, and a complete test example.
This guide walks through configuring ShardingSphere with Oracle in a Spring Boot‑MyBatis‑JPA project, shows custom sharding algorithms, entity/DAO/service code, and details the step‑by‑step debugging that uncovered schema handling bugs and version‑specific quirks, ending with a working solution.
This article explains how, in a Spring Boot 2.2.13 environment with ShardingSphere 4.0.1, enabling data encryption causes the expected ShardingDataSource bean not to be created, walks through the configuration and source code analysis, and provides a custom bean solution to resolve the issue.
An in‑depth interview with Zhang Liang, VP and founder of Apache ShardingSphere, explores the project’s three development phases, its graduation to a top‑level Apache project, the three‑layer plug‑in architecture, and practical lessons for building sustainable open‑source communities and enterprise‑grade database middleware.
The 2020 China Open‑Source Report highlights ShardingSphere as the top‑ranked Chinese‑led Apache project, detailing its distributed database middleware architecture, impressive community activity, broad industry adoption, and strategic vision for evolving database ecosystems.
This article explains the fundamentals of SQL parsing, covering lexer and parser mechanisms, context‑free grammar concepts, LL(1) and LR(1) parsing techniques, and demonstrates how Apache ShardingSphere implements a flexible, multi‑dialect SQL parser with features such as formatting, parameterization, and extensibility.
This article introduces Apache ShardingSphere's architecture—including ShardingSphere‑JDBC, ShardingSphere‑Proxy, and the upcoming Sidecar—explains data sharding principles, read/write splitting, and provides a complete Spring Boot example with Maven dependencies, Java source code, and configuration files for horizontal database partitioning.
This article provides a comprehensive technical analysis of ShardingSphere's XA distributed transaction capabilities, covering the CAP theory, X/Open DTP model, MySQL XA support, JTA mapping, Atomikos initialization, transaction lifecycle (begin, getConnection, commit, rollback), and the recovery mechanism with detailed code snippets and diagrams.
This article explains how Apache ShardingSphere’s Encrypt‑JDBC component provides a transparent, low‑cost data‑masking solution for both new and existing applications, detailing the architecture, configuration steps, migration process, limitations, and practical use‑case scenarios.
This article explains how to use ShardingSphere‑JDBC to achieve horizontal database sharding and read/write splitting in a Spring Boot application, covering ShardingSphere concepts, configuration, entity and DAO code, custom sharding algorithm, and practical testing with multiple MySQL instances.
This talk walks through Sharding-Proxy’s role within Apache ShardingSphere, detailing its architecture, SQL lifecycle from protocol decoding to routing, rewriting, execution, and result merging, while highlighting core principles, performance bottlenecks, and practical optimization techniques for high‑throughput, low‑latency database proxying.
This article introduces JDTX, a novel distributed transaction middleware that combines ACID guarantees, high performance, and 1PC architecture, explains its design principles, compares two‑phase commit with flexible transactions, details its MVCC and WAL implementation, and outlines integration with Apache ShardingSphere and future roadmap.
This article explains the design and implementation of JDTX, a distributed transaction middleware that combines MVCC, WAL, and a one‑phase commit model to provide strong ACID guarantees, high performance, and multi‑database support while outlining its challenges and future roadmap.
The article recounts JD Digits architect Zhang Liang’s experience representing the Apache ShardingSphere community at ApacheCon 2019 in Las Vegas, describing the conference atmosphere, community interactions, ShardingSphere’s observability talk and Shark Tank showcase, and the growing Chinese contribution to the Apache ecosystem.
This article explains how to use Apache ShardingSphere's data‑masking (encryption) features to protect both new and existing MySQL/Oracle/PostgreSQL/SQLServer workloads, providing detailed YAML configurations, migration steps, advantages, applicable scenarios, and limitations for secure database operations.
Apache ShardingSphere provides a complete, transparent, low‑cost data masking solution that lets both new and existing applications encrypt sensitive fields without modifying business SQL, using Encrypt‑JDBC or Encrypt‑Proxy, configurable encryption rules, and step‑by‑step migration guidance for seamless, secure database transformations.
This article explains how Apache ShardingSphere provides a comprehensive, transparent, and low‑cost data masking solution for both new and existing applications, detailing the background, use‑case scenarios, architecture, masking rules, and processing flow that enable secure encryption and decryption without modifying business logic.
This article explains how to implement database read‑write splitting in a Spring Boot application using ShardingSphere, covering the creation of master‑slave MySQL databases, Maven dependency setup, multi‑data‑source configuration, load‑balance algorithms, and handling replication lag with HintManager.
Apache ShardingSphere provides an integrated distributed transaction solution combining traditional ACID‑compliant two‑phase commit, flexible Saga‑based transactions, and automatic transaction management, detailing their architectures, features, trade‑offs, and usage guidelines for developers handling multi‑database microservice environments.
This article recounts the author’s year‑long experience transforming Sharding‑JDBC into the ShardingSphere ecosystem, detailing technical challenges, community building, Apache Incubator admission, and the project’s evolution into a financial‑grade distributed database solution.
This article explains the challenges of preserving ACID properties in distributed environments, introduces CAP and BASE trade‑offs, reviews industry transaction models such as XA, 2PC, weak XA, BED, TCC and Saga, and details how ShardingSphere implements and plans to evolve support for these distributed transaction mechanisms.
This article provides a comprehensive technical overview of Sharding-Proxy, covering its transparent MySQL‑protocol design, internal architecture, PreparedStatement optimization, HikariCP tuning, result‑set streaming, memory‑strict and connection‑strict modes, and practical configuration tips for high‑performance database proxying.
