Rb (Redis Blaster): Python Library for Non‑Replicated Redis Sharding

Rb (Redis Blaster) implements non‑replicated sharding for Redis as a Python library that sits on top of the standard python‑redis client. It introduces a custom routing system so that key‑value operations are automatically directed to the appropriate Redis server, eliminating the need for manual routing.

Installation

The library is published on PyPI and can be installed with a single command:

$ pip install rb

Configuration

Creating a cluster is straightforward. Provide a dictionary mapping host IDs to connection parameters and optional default values for all hosts:

from rb import Cluster

cluster = Cluster(hosts={
    0: {'port': 6379},
    1: {'port': 6380},
    2: {'port': 6381},
    3: {'port': 6382},
    4: {'port': 6379},
    5: {'port': 6380},
    6: {'port': 6381},
    7: {'port': 6382},
}, host_defaults={'host': '127.0.0.1'})

The hosts dictionary maps integer host IDs to their configuration; host_defaults supplies common values such as the hostname.

Routing

After the cluster is built, a routing client can be obtained via cluster.get_routing_client(). The client automatically routes each command to the correct node based on the key:

client = cluster.get_routing_client()
results = {}
for key in keys_to_look_up:
    results[key] = client.get(key)

The routing client behaves like a standard StrictClient but only supports commands that involve a single key. By default it batches compatible commands (e.g., multiple GET become a single MGET); this behavior can be disabled by setting auto_batch=False for more accurate debugging with MONITOR.

Parallel Execution

Using the map context manager, commands are sent in parallel and the results are Promise objects that resolve when the operation completes:

results = {}
with cluster.map() as client:
    for key in keys_to_fetch:
        results[key] = client.get(key)
for key, promise in results.iteritems():
    print('%s => %s' % (key, promise.value))

Similarly, cluster.all() or cluster.fanout() can be used to broadcast a command (e.g., flushdb()) to all hosts.

API Overview

The public API mirrors the underlying python‑redis library but adds cluster‑aware methods: add_host(...): add a host (mainly for unit tests). all(...): fan‑out to all hosts. execute_commands(mapping, *args, **kwargs): run a series of commands on the appropriate nodes, returning a mapping of Promise objects. fanout(hosts=None, ...): obtain a context manager that sends commands to a manually specified host list. get_local_client(host_id) and get_local_client_for_key(key): retrieve a standard Redis client for a specific host or the host determined by a key. get_router(): access the underlying router instance (e.g., PartitionRouter or ConsistentHashingRouter). get_routing_client(auto_batch=True): the main entry point for key‑based routing.

Clients

Two primary client classes are provided: RoutingClient: routes each command to a single target host. MappingClient: behaves like a Redis pipeline, executing commands in parallel and requiring a join() to collect results.

Routers

The routing logic is encapsulated in router classes derived from BaseRouter. Two concrete routers are shipped: PartitionRouter: uses crc32 % node_count to map a key to a host. ConsistentHashingRouter: applies a consistent‑hash algorithm, also requiring a contiguous host ID range.

Custom routers can be implemented by subclassing BaseRouter and overriding get_host_for_key and related methods.

Promise

The library defines a Promise class that mimics the ES6 Promise API, providing methods such as then, done, resolve, reject, and state flags ( is_pending, is_resolved, is_rejected). Promises are used to represent the eventual result of parallel operations.

Testing Utilities

For integration tests, rb.testing.TestSetup can spin up multiple Redis server processes, and rb.testing.make_test_cluster() creates a temporary cluster within a context manager, automatically cleaning up after use.

Overall, Rb provides a full‑featured, Pythonic way to work with sharded Redis deployments, handling routing, parallelism, and cluster management without requiring Redis replication.