Redis Pipeline, Transactions, Lua, Distributed Locks, Streams & Data Types

Advanced Redis Techniques Overview

Redis can serve as more than a simple cache. It provides a set of primitives that solve high‑frequency engineering problems such as bulk command efficiency, atomic multi‑step operations, cross‑instance mutual exclusion, reliable asynchronous processing, and large‑scale statistical calculations.

1. Pipeline – Reducing Network Round‑Trips

Pipeline batches commands on the client side and sends them in a single TCP write. It improves throughput but does not guarantee atomicity.

# Benchmark with 16 commands per request
redis-benchmark -h 127.0.0.1 -p 6379 -a 123456 -t set -n 100000 -P 16 -q

Java (Jedis) example

Jedis jedis = new Jedis("127.0.0.1", 6379);
Pipeline pipeline = jedis.pipelined();
for (int i = 1; i <= 1000; i++) {
    pipeline.set("pipeline:user:" + i, "user-" + i);
}
List<Object> results = pipeline.syncAndReturnAll();
System.out.println("result size = " + results.size());

Spring Data Redis example

@Service
public class ProductCachePipelineService {
    private final StringRedisTemplate stringRedisTemplate;
    public ProductCachePipelineService(StringRedisTemplate stringRedisTemplate) {
        this.stringRedisTemplate = stringRedisTemplate;
    }
    public List<Object> batchSetProductCache(List<Long> productIds) {
        return stringRedisTemplate.executePipelined(connection -> {
            for (Long productId : productIds) {
                String key = "product:" + productId;
                String value = "{\"id\":" + productId + ",\"name\":\"demo\"}";
                connection.stringCommands().set(key.getBytes(StandardCharsets.UTF_8),
                                             value.getBytes(StandardCharsets.UTF_8));
            }
            return null; // result collected by executePipelined
        });
    }
}

Key considerations

Pipeline does not roll back on errors – successful commands remain.

Result order matches command order.

Do not use for commands that depend on previous results.

Batch size should be tuned (e.g., 100‑1000 commands) to avoid excessive client memory or server output‑buffer pressure.

2. Transaction vs. Lua – When to Use Which

2.1 Redis Transaction (MULTI/EXEC)

A transaction queues commands and executes them sequentially without interleaving other clients. It guarantees ordering but not full atomic rollback.

# Transaction example (CLI)
MULTI
SET account:A 100
INCR account:B
EXEC

Java (Jedis) example

Jedis jedis = pool.getResource();
Transaction tx = jedis.multi();
tx.set("order:1001:status", "paying");
tx.incr("order:1001:version");
List<Object> results = tx.exec();
System.out.println("SET result = " + results.get(0));
System.out.println("INCR result = " + results.get(1));

Spring Data Redis example

@Service
public class RedisTransactionService {
    private final StringRedisTemplate template;
    public RedisTransactionService(StringRedisTemplate template) { this.template = template; }
    public List<Object> createPayingOrder(String orderId) {
        return template.execute(new SessionCallback<List<Object>>() {
            @Override
            public List<Object> execute(RedisOperations ops) {
                ops.multi();
                ops.opsForValue().set("order:" + orderId + ":status", "paying");
                ops.opsForValue().increment("order:" + orderId + ":version");
                return ops.exec();
            }
        });
    }
}

When to choose

Simple ordered batch where you only need to avoid interleaving (e.g., initializing a set of keys).

Do not rely on it for complex conditional logic – a failing command does not undo previous ones.

2.2 Lua – True Atomic Multi‑Step Logic

Lua scripts are executed as a single command, guaranteeing atomicity for the whole script.

# Stock deduction script (CLI)
EVAL "if tonumber(redis.call('GET', KEYS[1])) > 0 then
    return redis.call('DECR', KEYS[1])
else
    return -1
end" 1 stock:1001

Java (Spring) example

String script = """
if tonumber(redis.call('GET', KEYS[1])) > 0 then
    return redis.call('DECR', KEYS[1])
else
    return -1
end
""";
DefaultRedisScript<Long> redisScript = new DefaultRedisScript<>();
redisScript.setScriptText(script);
redisScript.setResultType(Long.class);
Long result = stringRedisTemplate.execute(redisScript,
    Collections.singletonList("stock:" + productId));

Key points

Keep scripts short; avoid long loops or heavy scans – they block the Redis main thread.

In a cluster, all keys used by the script must belong to the same hash slot (use hash tags like {order}:stock).

For high‑frequency scripts, load them once with SCRIPT LOAD and invoke via EVALSHA to save bandwidth.

3. Distributed Lock – Cross‑Instance Mutual Exclusion

Redis does not have a built‑in lock primitive; the safe pattern uses SET key value NX EX seconds for acquisition and a Lua script for release.

# Acquire lock (CLI)
SET lock:order:1001 request-uuid-001 NX EX 30
# Release lock (Lua)
EVAL "if redis.call('GET', KEYS[1]) == ARGV[1] then
    return redis.call('DEL', KEYS[1])
else
    return 0
end" 1 lock:order:1001 request-uuid-001

Jedis implementation

public String tryLock(String key, int ttlSeconds) {
    String token = UUID.randomUUID().toString();
    String result = jedis.set(key, token, SetParams.setParams().nx().ex(ttlSeconds));
    return "OK".equals(result) ? token : null;
}
public boolean unlock(String key, String token) {
    String lua = "if redis.call('GET', KEYS[1]) == ARGV[1] then " +
                 "return redis.call('DEL', KEYS[1]) else return 0 end";
    Object res = jedis.eval(lua, Collections.singletonList(key),
                           Collections.singletonList(token));
    return Long.valueOf(1).equals(res);
}

Spring Data Redis implementation

public String tryLock(String key, Duration ttl) {
    String token = UUID.randomUUID().toString();
    Boolean ok = stringRedisTemplate.opsForValue()
        .setIfAbsent(key, token, ttl);
    return Boolean.TRUE.equals(ok) ? token : null;
}
private static final String UNLOCK_LUA = """
if redis.call('GET', KEYS[1]) == ARGV[1] then
    return redis.call('DEL', KEYS[1])
else
    return 0
end
""";
public boolean unlock(String key, String token) {
    Long result = stringRedisTemplate.execute(
        new DefaultRedisScript<>(UNLOCK_LUA, Long.class),
        Collections.singletonList(key), token);
    return Long.valueOf(1).equals(result);
}

Redisson (recommended for production)

@Service
public class OrderLockService {
    private final RedissonClient client;
    public OrderLockService(RedissonClient client) { this.client = client; }
    public boolean handleOrder(String orderId) throws InterruptedException {
        RLock lock = client.getLock("lock:order:" + orderId);
        // wait up to 3 seconds, auto‑renewal (watchdog) enabled
        if (!lock.tryLock(3, TimeUnit.SECONDS)) return false;
        try {
            // critical section
            System.out.println("process order " + orderId);
            return true;
        } finally {
            if (lock.isHeldByCurrentThread()) lock.unlock();
        }
    }
}

When to use which implementation

Learning / low‑traffic: raw Jedis (full control).

Typical Spring Boot service with occasional lock: Spring Redis + Lua.

High‑traffic production: Redisson (re‑entrancy, watchdog, fair lock, read/write lock, etc.).

Lock suitability checklist

Multiple service instances need mutual exclusion.

Operation cannot be expressed by a single atomic Redis command or DB unique constraint.

Critical section is short (seconds).

Business logic is idempotent (DB unique index, status machine, version field).

4. Messaging – List, Pub/Sub, Stream

Redis can act as a lightweight message queue. Choose based on durability, consumer‑group support, and acknowledgment needs.

4.1 List – Simple Blocking Queue

# Producer
LPUSH mq:order order-1001
# Consumer (blocks up to 5 s)
BRPOP mq:order 5

Pros: trivial API, works for single‑consumer scenarios. Cons: no ACK, message loss if consumer crashes after BRPOP.

4.2 Pub/Sub – Real‑time Broadcast

# Subscriber
SUBSCRIBE channel:notice
# Publisher
PUBLISH channel:notice "hello"

Pros: push‑style delivery, zero latency. Cons: no persistence, offline subscribers miss messages.

4.3 Stream – Reliable Queue

# Create consumer group (auto‑creates stream if missing)
XGROUP CREATE stream:order g1 0 MKSTREAM
# Add a message
XADD stream:order * orderId 1001 type paid
# Consumer reads (blocks 2 s, max 1 message)
XREADGROUP GROUP g1 c1 COUNT 1 BLOCK 2000 STREAMS stream:order >
# After processing, acknowledge
XACK stream:order g1 1718000000000-0
# Inspect pending entries
XPENDING stream:order g1

Pros: persisted, consumer groups, explicit ACK, pending‑list for retries. Cons: higher memory footprint than List; not a full‑featured MQ (no topic routing, limited clustering).

Java usage examples

Jedis : use lpush/brpop, publish/subscribe, and xadd/xreadGroup/xack directly.

Spring Data Redis : opsForList(), convertAndSend(), and opsForStream() with createGroup, read, acknowledge.

Redisson : RBlockingQueue, RTopic, RStream with high‑level APIs.

Selection guidance

Simple background jobs, low reliability → List.

Live notifications, chat, config reload → Pub/Sub.

Order processing, SMS, email where loss is unacceptable → Stream (or a dedicated MQ for very high scale).

5. Specialized Data Types

5.1 BitMap – One bit per boolean state

# User 1001 signs in on day 4 of July 2026 (offset 3)
SETBIT sign:1001:202607 3 1
# Check
GETBIT sign:1001:202607 3   # returns 1
# Count signed days in the month
BITCOUNT sign:1001:202607   # returns number of 1‑bits
# Memory usage (should be a few bytes)
MEMORY USAGE sign:1001:202607

Typical use: daily sign‑in, activity flags, feature toggles. Memory cost is ~1 bit per user per day (≈4 bytes per user per month).

Redisson wrapper

RBitSet bits = redisson.getBitSet("sign:" + userId + ":202607");
bits.set(dayOfMonth - 1, true); // set
boolean signed = bits.get(dayOfMonth - 1);
long signedDays = bits.cardinality();

5.2 HyperLogLog – Approximate distinct count (≈0.81 % error)

# Add visitors (duplicates are ignored automatically)
PFADD uv:20260704 user1 user2 user3 user1
# Estimate UV
PFCOUNT uv:20260704   # e.g., 3
# Merge several days into a weekly estimate
PFMERGE uv:week uv:20260701 uv:20260702 uv:20260703 uv:20260704

Use case: daily active users, unique IP count, any scenario where an approximate cardinality is acceptable.

Redisson wrapper

RHyperLogLog<String> uv = redisson.getHyperLogLog("uv:" + date);
uv.add(userId);
long estimate = uv.count();

5.3 Bloom Filter – Fast existence pre‑check

# Create a filter for 1 M items with 1 % false‑positive rate
BF.RESERVE bf:product 0.01 1000000
# Add a product id
BF.ADD bf:product 1001
# Test existence
BF.EXISTS bf:product 1001   # 1 (definitely exists)
BF.EXISTS bf:product 9999   # 0 (definitely not)
# Check filter stats
BF.INFO bf:product

Typical use: cache‑miss protection, blacklist, “does this id possibly exist?” before hitting DB.

Redisson wrapper

RBloomFilter<Long> filter = redisson.getBloomFilter("bf:product");
filter.tryInit(1_000_000L, 0.01); // once at startup
filter.add(productId);
boolean maybeExists = filter.contains(productId); // false → definitely not

5.4 GEO – Radius queries on latitude/longitude

# Add two shops (lon lat name)
GEOADD shop:geo 116.40 39.90 shopA 116.41 39.91 shopB
# Distance between them (km)
GEODIST shop:geo shopA shopB km   # 1.4663
# Find shops within 5 km of a point
GEOSEARCH shop:geo FROMLONLAT 116.40 39.90 BYRADIUS 5 km ASC WITHDIST COUNT 10

Use case: nearby store/driver lookup, location‑based recommendations. Remember: longitude first, then latitude.

6. Practical E‑commerce Mapping

Bulk product cache warm‑up → Pipeline .

Product‑id existence check → Bloom Filter (fast reject).

Flash‑sale stock deduction → Lua script (atomic check‑and‑decrement).

Prevent duplicate order creation → database unique index (lock optional for extra safety).

Async SMS after order → Stream (reliable, ACK, retry).

Daily UV → HyperLogLog (≈12 KB per day).

User daily sign‑in → BitMap (≈4 bytes per user per month).

Nearby store search → GEO radius query.

Rule of thumb: use Redis for speed, deduplication, and lightweight coordination; keep the ultimate source of truth in the database (unique constraints, status fields, version numbers).

7. Production Checklist & Common Pitfalls

Verification commands

# Slow commands (look for EVAL/EVALSHA)
SLOWLOG GET 10
# Client list – spot blocked connections
CLIENT LIST
# Stream pending entries (un‑ACKed messages)
XPENDING stream:order g1
# Memory usage per key
MEMORY USAGE sign:1001:202607
# Command statistics
INFO commandstats

Typical mistakes and fixes

Thinking Pipeline = atomic transaction – It only reduces network latency. Use Lua for true atomicity.

Assuming Redis transaction rolls back on error – Only the failing command is reported; earlier commands stay.

Using DEL to release a lock – May delete another client’s lock. Always release with a Lua script that checks the token.

Specifying leaseTime in Redisson – The watchdog stops renewing; only use when you know the exact max execution time.

Using Pub/Sub for reliable delivery – Messages are lost for offline subscribers. Switch to Stream or a dedicated MQ.

Relying on HyperLogLog for exact counts – It provides an estimate. Use a Set or DB query when precision is required.

Treating Bloom Filter “exists” as certainty – It can return false positives. Follow up with a real data lookup.

Running long loops inside Lua – Blocks the Redis thread. Keep scripts short and side‑effect free.

Mis‑ordering longitude/latitude in GEO commands – Leads to wrong distances. Always lon lat.

By following the guidelines above, you can confidently pick the right Redis primitive, implement it safely, and troubleshoot issues in production.