Is Sharding Dead? A Technical Comparison with NewSQL Databases

Source: Java技术指北

Recent discussions with peers often raise the question of whether to adopt middleware‑based sharding (分库分表) or a NewSQL distributed database. This article compares the two approaches by analyzing their core characteristics, implementation principles, advantages, drawbacks, and suitable scenarios.

What Makes NewSQL Advanced?

The paper pavlo-newsql-sigmodrec classifies Spanner, TiDB, and OceanBase as first‑generation NewSQL architectures, while middleware solutions such as Sharding‑Sphere, Mycat, and DRDS belong to a second generation. The third generation (cloud databases) is not covered here.

Using middleware (SDK or proxy) with traditional relational databases does constitute a distributed architecture because storage is also distributed and horizontal scaling is possible.

Is It a “Pseudo” Distributed Database?

From an architectural perspective, middleware adds redundant SQL parsing and execution‑plan generation on top of the underlying DB, which is inefficient compared with native distributed databases.

Distributed Transactions

NewSQL databases inherit the challenges of distributed transactions. The CAP theorem still applies; they do not magically break it. Google Spanner, which many NewSQL systems emulate, claims CA‑like behavior by relying on a private global network and strong operational practices.

Two‑phase commit (2PC) is still used, but implementations such as Google Percolator combine a Timestamp Oracle, MVCC, Snapshot Isolation, and primary/secondary locks to reduce latency. However, SI can cause high abort rates on hot data and may exhibit write skew.

Compared with XA, 2PC adds overhead (GID acquisition, network latency, log persistence), which becomes significant in high‑concurrency scenarios such as batch banking transactions.

Spanner’s Distributed Transaction Benchmarks

Although NewSQL products advertise full support for distributed transactions, best‑practice guides still recommend minimizing their use because of the performance cost.

When strong consistency is too expensive, flexible (BASE) models—Saga, TCC, reliable messaging—are suggested for large‑scale OLTP workloads.

HA and Multi‑Region Active‑Active

Traditional master‑slave replication, even semi‑synchronous, can lose data under extreme conditions. Modern solutions adopt Paxos or Raft (e.g., Google Spanner, TiDB, CockroachDB, OceanBase) to achieve high availability with majority‑write guarantees.

These protocols can also be applied to MySQL (e.g., MySQL Group Cluster), suggesting that master‑slave may soon become obsolete.

Implementing true multi‑region active‑active requires low network latency; otherwise, the added commit latency is unacceptable for OLTP systems.

Scale and Sharding Mechanisms

While Paxos solves availability, horizontal scaling still relies on sharding. NewSQL databases embed automatic sharding, hotspot detection, and region splitting (e.g., TiDB splits a region at 64 MiB).

In contrast, sharding‑plus‑middleware forces developers to design split keys, routing rules, and expansion procedures upfront, increasing complexity.

Uniform sharding strategies may not align with domain models, leading to distributed transactions—for example, banking workloads that write customer, account, and transaction tables together may span multiple shards when using range‑based sharding.

Distributed SQL Support

Both approaches handle single‑shard SQL well. NewSQL databases, being general‑purpose, support cross‑shard joins, aggregations, and complex queries via cost‑based optimization (CBO) using distributed statistics.

Middleware solutions typically rely on rule‑based optimization (RBO) and lack the metadata needed for efficient cross‑shard execution, limiting their support for joins and aggregations.

Storage Engine

Traditional RDBMS engines are disk‑oriented B+‑tree structures, optimizing random reads but suffering from random writes. NewSQL engines often use LSM trees, converting random writes into sequential writes for higher write throughput, though read performance may be lower unless mitigated by SSDs, bloom filters, etc.

Maturity and Ecosystem

Evaluating distributed databases requires multi‑dimensional testing: development status, community, monitoring tools, feature coverage, DBA talent, SQL compatibility, performance, HA, online scaling, transaction support, isolation levels, and online DDL.

NewSQL products have matured in internet‑scale environments but still lag behind decades‑old relational databases in overall stability, ecosystem, and compatibility with legacy systems.

Guidance for Choosing Between Sharding and NewSQL

Consider the following questions: need for strong consistency, data growth predictability, scaling frequency, throughput vs latency priority, application transparency, and availability of DBA expertise.

If two or three answers are affirmative, NewSQL may be worth the learning cost, especially for fast‑growing internet services. Otherwise, sharding with middleware offers a lower‑risk, lower‑cost solution that reuses existing relational ecosystems.

Conclusion

Sharding is unlikely to disappear soon; it remains a viable, lower‑cost option for many traditional industries. NewSQL provides a richer feature set but carries higher operational complexity and may not yet be fully mature for mission‑critical core systems.