Mastering Twemproxy: A Fast Redis/Memcached Proxy for Scalable Caching

1. Twemproxy Overview

When handling large numbers of Redis or Memcached instances, client‑side sharding algorithms such as consistent hashing are often used to achieve clustering. Before Redis Cluster became production‑ready, many deployments used a proxy layer to provide clustered storage.

Twitter, which runs one of the world’s largest Redis clusters, open‑sourced Twemproxy (also called nutcracker) as a Redis/Memcached proxy.

Twemproxy is a fast single‑threaded proxy written in C, supporting the Memcached ASCII protocol and newer Redis protocol. It is licensed under Apache 2.0 and runs on Linux (it cannot be compiled on macOS because it depends on the epoll API).

By inserting a proxy layer, Twemproxy unifies management and distribution of multiple backend Redis or Memcached instances, allowing applications to interact only with Twemproxy without needing to know the number or location of the actual storage nodes.

2. Twemproxy Features

Automatic removal of failed nodes

Configurable reconnection delay

Node removal after a configurable number of failures

Suitable for cache storage

HashTag support

Allows custom mapping of two keys to the same backend instance

Reduces direct connections to Redis

Maintains persistent connections

Configurable number of connections per backend

Automatic sharding across multiple Redis instances

Multiple hash algorithms, including consistent hashing

Configurable backend instance weights

Avoids single‑point failures

Multiple proxy instances can be deployed in parallel; clients automatically select an available one

Supports Redis pipelining

Streaming and batch request handling to reduce round‑trip overhead

State monitoring

Configurable monitoring IP/port returning JSON status

Adjustable monitoring refresh interval

High throughput

Connection and memory reuse

Aggregates multiple requests into Redis pipelining

Twemproxy can also be extended by modifying Redis source code to extract a middle‑layer proxy; its performance benefits from Linux epoll for high concurrency.

3. Limitations

Does not support multi‑key operations such as set intersections/unions (except MGET and DEL)

No support for Redis transactions

Error messages are not very detailed

Does not support the SELECT command

4. Installation and Configuration

Installation steps (see the GitHub repository https://github.com/twitter/twemproxy ):

apt-get install automake
apt-get install libtool
git clone git://github.com/twitter/twemproxy.git
cd twemproxy
autoreconf -fvi
./configure --enable-debug=log
make
src/nutcracker -h

Typical configuration (example):

redis1:
  listen: 127.0.0.1:6379   # port for Twemproxy
  redis: true               # proxying Redis
  hash: fnv1a_64            # hash function
  distribution: ketama      # hash algorithm
  auto_eject_hosts: true    # eject unresponsive nodes
  timeout: 400              # ms
  server_retry_timeout: 2000
  server_failure_limit: 1
  servers:
    - 127.0.0.1:6380:1
    - 127.0.0.1:6381:1
    - 127.0.0.1:6382:1

redis2:
  listen: 0.0.0.0:10000
  redis: true
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: false
  timeout: 400
  servers:
    - 127.0.0.1:6379:1
    - 127.0.0.1:6380:1
    - 127.0.0.1:6381:1
    - 127.0.0.1:6382:1

Multiple Twemproxy instances can be run simultaneously, each handling read/write operations, allowing applications to completely avoid single‑point failures.