How ShardingSphere-JDBC Implements Transparent Sharding with Dynamic Proxies

ShardingSphere is an Apache‑hosted distributed database middleware, and its ShardingSphere-JDBC module deeply adapts the JDBC standard interface. By combining dynamic proxies, a strategy engine, and an SPI extension mechanism, it achieves fully JDBC‑compatible sharding. This article is based on ShardingSphere-JDBC 5.3.0 source code (GitHub) and provides an in‑depth analysis of its underlying implementation with key code examples.

1. JDBC Specification and ShardingSphere Compatibility Design

1. JDBC Core Interface Standardization

The standardized JDBC interfaces such as DataSource, Connection, and Statement form the basis for ShardingSphere's compatibility. ShardingSphere injects sharding logic via interface adaptation plus dynamic proxy without modifying the original interface definitions.

2. Modular Architecture

The core modules of ShardingSphere-JDBC include:

Kernel : responsible for SPI loading, configuration management, and context maintenance.

Sharding : handles SQL parsing, routing, rewriting, execution, and result merging.

Transaction : supports local transactions, XA, TCC and other distributed transaction modes.

Adapter : provides a complete JDBC implementation compatible with protocols such as MySQL and PostgreSQL.

2. Core Implementation of the Adapter Pattern

1. DataSource Adapter

ShardingSphere provides ShardingSphereDataSource, which extends AbstractDataSourceAdapter. Its core logic is shown below:

public class ShardingSphereDataSource extends AbstractDataSourceAdapter {
    public Connection getConnection() throws SQLException {
        return new ShardingSphereConnection(this);
    }
}

Dynamic Proxy : ShardingSphereDataSource uses a Proxy class to generate a proxy object that forwards JDBC method calls to the sharding engine.

SPI Loading : ServiceLoader loads sharding rules, data source configurations, etc.

2. Connection Adapter

ShardingSphereConnection

implements the Connection interface. Its main responsibilities include connection management, transaction synchronization, and method interception.

public class ShardingSphereConnection implements Connection {
    private final Map<String, Connection> physicalConnections;
    public PreparedStatement prepareStatement(String sql) throws SQLException {
        Collection<String> targetDataSources = route(sql);
        List<Connection> connections = getConnections(targetDataSources);
        return new ShardingSpherePreparedStatement(this, sql, connections);
    }
}

Connection Management : maintains a Map<String, Connection> of physical connections.

Transaction Synchronization : uses TransactionManager to synchronize transaction states across shards.

Method Interception : intercepts methods like setAutoCommit and setReadOnly to ensure consistency across multiple connections.

3. Statement Adapter

ShardingSpherePreparedStatement

implements PreparedStatement. Its core logic includes SQL parsing and routing, sharding execution, and result merging.

public class ShardingSpherePreparedStatement implements PreparedStatement {
    public ResultSet executeQuery() throws SQLException {
        List<ResultSet> resultSets = new ArrayList<>();
        for (Connection connection : physicalConnections) {
            PreparedStatement stmt = connection.prepareStatement(sql);
            ResultSet rs = stmt.executeQuery();
            resultSets.add(rs);
        }
        return new ShardingSphereResultSet(resultSets);
    }
}

SQL Parsing & Routing : calls SQLRouter to parse the sharding key (e.g., user_id) and determine target shards.

Sharding Execution : dispatches the SQL to multiple shard nodes and generates a ShardingSphereResultSet.

Result Merging : uses MergeEngine to combine result sets from different shards.

4. ResultSet Adapter

ShardingSphereResultSet

implements ResultSet and merges multiple shard result sets.

public class ShardingSphereResultSet implements ResultSet {
    private final List<ResultSet> physicalResultSets;
    private int currentResultSetIndex = 0;
    public boolean next() throws SQLException {
        while (currentResultSetIndex < physicalResultSets.size()) {
            ResultSet current = physicalResultSets.get(currentResultSetIndex);
            if (current.next()) {
                return true;
            }
            currentResultSetIndex++;
        }
        return false;
    }
}

Streaming Merge : reads rows sequentially to avoid memory overflow.

Sorting & Pagination : uses OrderByMergeEngine or PaginationMergeEngine for global ordering and paging.

3. Core Components of Sharding Logic

1. SQL Parsing and Routing

ShardingSphere uses ANTLR4 to parse the SQL AST and extract sharding keys (e.g., t_order.user_id). Routing rules are implemented by ShardingAlgorithm, supporting standard sharding (hash or range), complex sharding (multiple keys), and broadcast tables.

Source example: standard sharding algorithm

public interface StandardShardingAlgorithm<T> extends ShardingAlgorithm {
    Collection<String> doSharding(Collection<String> availableTargetNames, StandardShardingValue<T> shardingValue);
}

public class HashModShardingAlgorithm implements StandardShardingAlgorithm<Integer> {
    @Override
    public Collection<String> doSharding(Collection<String> availableTargetNames, StandardShardingValue<Integer> shardingValue) {
        int shardCount = availableTargetNames.size();
        int shardIndex = shardingValue.getValue() % shardCount;
        return Collections.singletonList(availableTargetNames.stream().skip(shardIndex).findFirst().orElseThrow());
    }
}

2. Distributed Transaction

ShardingSphere supports the following transaction modes:

Local Transaction : suitable for single‑shard scenarios.

XA Transaction : implemented via Atomikos or Narayana for cross‑shard transactions.

TCC Transaction : a flexible two‑phase commit suitable for high‑concurrency scenarios.

Source example: XA transaction manager

public class XATransactionManager implements TransactionManager {
    public void begin() throws SQLException {
        // Start XA transaction
        xaResource.start(xid, TM_BEGIN);
    }
    public void commit() throws SQLException {
        // Commit XA transaction
        xaResource.end(xid, TMSUCCESS);
        xaResource.prepare(xid);
        xaResource.commit(xid, false);
    }
}

3. Elastic Scaling

ShardingSphere 5.x introduces elastic data migration, enabling online scaling (adding or removing shard nodes) without downtime.

4. Performance Optimization and Extensibility Design

1. Connection Pool Management

ShardingSphere manages physical connection pools via HikariCP or Druid. ShardingSphereConnection reuses existing connections to avoid repeated creation.

Source example: connection pool configuration

# config-sharding.yaml
dataSources:
  ds_0:
    driver-class-name: com.mysql.cj.jdbc.Driver
    url: jdbc:mysql://localhost:3306/ds_0
    username: root
    password: root
    connectionPool:
      maxPoolSize: 10

2. Result Set Merging Optimization

Streaming Merge : avoids loading all shard results into memory at once.

Parallel Merge : reads shard results concurrently using multiple threads to improve performance.

3. SPI Extension Mechanism

ShardingSphere’s SPI mechanism allows developers to plug in custom sharding algorithms, transaction managers, etc. Example of a custom sharding algorithm:

public class CustomShardingAlgorithm implements StandardShardingAlgorithm {
    @Override
    public Collection<String> doSharding(Collection<String> availableTargetNames, ShardingValue shardingValue) {
        // Custom sharding logic
        return availableTargetNames.stream()
            .filter(name -> name.contains(shardingValue.getValue().toString()))
            .collect(Collectors.toList());
    }
}

5. Summary

ShardingSphere-JDBC 5.x achieves full JDBC‑compatible sharding through dynamic proxies, the adapter pattern, and a strategy engine . Key design highlights include:

Full JDBC Compatibility : developers can enable sharding without changing existing code.

Modular Architecture : sharding, transaction, and elastic scaling functionalities are decoupled for easy extension.

High‑Performance Optimizations : connection pool reuse, streaming result set merging, and parallel processing reduce resource consumption.

Flexible Extensibility : SPI support enables custom sharding algorithms, transaction modes, and other plug‑ins.

As ShardingSphere evolves, its capabilities in the distributed database middleware space continue to strengthen, making it a vital tool for building high‑performance, highly available distributed systems.