What Hidden Pitfalls Await in Distributed Systems? A Deep Dive into Messaging, Caching, and Sharding

Introduction

Distributed systems are a hot interview topic, but many developers still wonder what they really are and why they matter.

Analogy with Naruto

Just as Naruto’s multiple shadow clones share experiences, distributed nodes share state and cooperate, yet they also consume extra resources ("chakra").

Core Concepts

Distributed systems consist of multiple independent computers that appear as a single service.

They enable workload splitting (macro) and service deployment across machines (micro).

CAP Theorem

In any distributed system you can only achieve two of the three guarantees:

Consistency : all nodes see the same latest data.

Availability : every request receives a response, though it may be stale.

Partition Tolerance : the system continues operating despite network partitions.

BASE Theory

BASE (Basically Available, Soft state, Eventually consistent) relaxes strict consistency to improve availability, allowing temporary inconsistency that eventually converges.

Message Queue Pitfalls

Non‑Idempotence

Repeated consumption can cause data inconsistency; solutions include unique IDs, database checks, or Redis SET which is naturally idempotent.

Message Loss

Producer loss: use transaction ( channel.txSelect) or confirm mode ( ack / nack).

Broker loss: enable durable queues and set deliveryMode=2.

Consumer loss: disable auto‑ack and ack only after processing.

Message Disorder

Ensure ordering by sending related messages to the same partition or queue and using single‑consumer processing per queue.

Backlog, Expiration, Full Queue

Backlog: scale consumers or increase queue count.

Expiration: monitor and re‑import expired messages.

Full queue: purge useless messages or accelerate consumption.

Distributed Cache Pitfalls (Redis)

Sentinel provides high availability but can lose data during failover.

Asynchronous replication may cause data loss if the master crashes before syncing.

Split‑brain scenarios lead to divergent masters and lost writes.

Mitigations

Configure min-slaves-to-write=1 and min-slaves-max-lag=10.

Sharding (Database Partitioning) Pitfalls

Horizontal vs. vertical splitting each has trade‑offs (cross‑db joins, consistency).

Unique ID generation is critical; options include UUID, timestamp, Snowflake, Baidu UIDGenerator, Meituan Leaf‑Snowflake.

Snowflake Example

64‑bit ID: 41‑bit timestamp, 10‑bit machine ID, 12‑bit sequence; provides trend‑increasing IDs but depends on accurate clocks.

Distributed Transaction Pitfalls

XA (two‑phase commit) – suitable for monoliths, not microservices.

TCC (try‑confirm‑cancel) – strong consistency but high implementation cost.

SAGA – compensating actions, high throughput, but no isolation.

Reliable message (prepared/commit) – uses MQ to coordinate.

Max‑effort notification – retries before manual compensation.

Choosing a Model

Payments: prefer TCC.

Large systems with looser consistency: SAGA or reliable message.

Simple monoliths: XA.

Conclusion

Distributed architectures bring both advantages and challenges; whether to adopt them depends on business needs, timeline, cost, and team expertise.