Understanding Redis Clustering: Client Sharding, Proxy Sharding, and Codis Solutions

1. Common Redis Cluster Techniques

Redis originally supports only a single instance with limited memory (10‑20 GB), which cannot meet the demands of large‑scale online services and leads to low resource utilization on servers with 100‑200 GB RAM.

To address insufficient single‑node capacity, many internet companies have built self‑service clustering solutions that physically shard data across multiple Redis instances, typically one instance per shard.

1.1 Client‑Side Sharding

This method places the sharding logic in the application code, which routes requests to the appropriate Redis nodes based on predefined rules. It avoids a middle‑layer proxy, offering better performance, but requires manual updates when nodes are added or removed, making operations cumbersome and unsuitable for small teams without strong DevOps support.

1.2 Proxy Sharding

A dedicated proxy (distributed middleware) receives client requests, applies routing rules, and forwards them to the correct Redis instance. Although it introduces a slight performance overhead, it simplifies application development and operations; Twemproxy is a widely used open‑source example.

1.3 Redis Cluster

Redis Cluster maps all keys to 16 384 slots, with each node responsible for a subset. Clients can query any node; if the data is not on that node, it redirects the request to the appropriate owner. This decentralized design is heavier than a single instance and is still not commonly adopted in production environments.

2. Twemproxy and Its Limitations

Twemproxy, an open‑source proxy sharding solution from Twitter, forwards requests to backend Redis servers according to routing rules and returns responses to the client. While it solves the capacity issue of a single Redis instance, Twemproxy itself is a single point of failure and requires external high‑availability solutions (e.g., Keepalived). Its major pain points include difficulty scaling (adding or removing Redis nodes) and a lack of an operational control panel.

3. Codis in Practice

Codis, open‑sourced by PeaPod in November 2014, is built with Go and C and has become a popular Chinese open‑source Redis clustering solution. It is used extensively in PeaPod’s own Redis workloads and has demonstrated stable performance in pressure tests, eventually achieving up to 100% faster throughput than Twemproxy under multi‑instance scenarios.

3.1 Architecture

Codis introduces the concept of a Group, which consists of one Redis master and at least one slave. This design enables automatic failover via a dashboard without modifying application configuration. Codis modifies the Redis server source to support hot rebalancing (Auto Rebalance) and uses ZooKeeper to store pre‑sharded slot information (1024 slots) and to maintain group metadata, providing distributed locks and other services.

3.2 Tips and Caveats

Seamless migration from Twemproxy is possible using the Codis‑port tool, which synchronizes data and requires only a configuration change.

Codis offers a Java client called Jodis that automatically detects failed proxies and routes around them, ensuring high availability for Java applications.

Pipeline support allows batching of requests for higher throughput.

Codis does not handle master‑slave data synchronization; operators must ensure consistency themselves, which keeps the system lightweight and easier to deploy in production.