Taming Message Storms: Redis 7.x Engineering Practices for Enterprise Live‑Streaming Platforms

Business background

A large live‑streaming platform runs >100k concurrent rooms, >30 million online users, and peak per‑room message rates of 20k‑80k msgs/s.

Message type classification

Real‑time broadcast (chat, likes) – ultra‑low latency.

Strongly consistent updates (gift ranking, points) – atomic.

Control messages (bans, announcements) – low latency, low loss tolerance.

Offline statistics – handled by streams or MQ.

Legacy architecture failures

Traditional Pub/Sub floods the cluster because PUBLISH/SUBSCRIBE is a global broadcast.

Lua scripts are scattered across services, causing version drift, cache misses and audit gaps.

Multiple teams share a cluster with root‑like permissions, leading to accidental deletions.

Transformation goals

Reduce chat P99 latency from 200 ms to < 50 ms.

Stably handle 50k msgs/s in a hot room.

Automatic subscription recovery after failover.

Enforce per‑service key‑level ACLs.

Redis 7.x core primitives

Sharded Pub/Sub

– channel‑level slot broadcasting; eliminates chat broadcast storms. Function – server‑side persistent function library; replaces scattered Lua with versioned, auditable logic. ACL v2 – fine‑grained user, command and key‑pattern permissions; turns a shared cluster into a multi‑tenant controlled base.

Sharded Pub/Sub in depth

Why the old Pub/Sub is inefficient

When a hot room emits 50 k messages per second, each PUBLISH is forwarded to every node, saturating intra‑cluster bandwidth, CPU and raising P99 latency.

Core design

Channel name is hashed to a slot like a key.

The primary node of that slot owns the channel. SPUBLISH sends only to the owning shard. SSUBSCRIBE subscribes to the shard channel; client libraries handle routing.

Channel naming guidelines

chat:{roomId}
gift_effect:{roomId}
ctrl:{roomId}

Benefits: clear semantics, alignment with business entities, easy secondary sharding for ultra‑hot rooms.

Secondary sharding for hot rooms

For a super‑hot room split the channel into sub‑channels:

chat:{10001}:0
chat:{10001}:1
chat:{10001}:2
chat:{10001}:3

Publish side hashes userId % 4 to select a sub‑channel; the gateway subscribes to all sub‑channels, keeping the client unaware of the split.

Production‑grade Java publisher

package com.example.live.redis;
import io.lettuce.core.RedisURI;
import io.lettuce.core.cluster.RedisClusterClient;
import io.lettuce.core.cluster.api.StatefulRedisClusterConnection;
import io.lettuce.core.cluster.api.async.RedisAdvancedClusterAsyncCommands;
import jakarta.annotation.PostConstruct;
import jakarta.annotation.PreDestroy;
import java.time.Duration;
import java.util.List;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;

@Slf4j
@Component
public class ShardedPubPublisher {
    private RedisClusterClient client;
    private StatefulRedisClusterConnection<String, String> connection;
    private RedisAdvancedClusterAsyncCommands<String, String> asyncCommands;

    @PostConstruct
    public void init() {
        List<RedisURI> nodes = List.of(
            RedisURI.Builder.redis("10.0.0.11", 7001).withTimeout(Duration.ofSeconds(2)).build(),
            RedisURI.Builder.redis("10.0.0.12", 7002).withTimeout(Duration.ofSeconds(2)).build(),
            RedisURI.Builder.redis("10.0.0.13", 7003).withTimeout(Duration.ofSeconds(2)).build()
        );
        client = RedisClusterClient.create(nodes);
        connection = client.connect();
        asyncCommands = connection.async();
    }

    public void publishChatMessage(long roomId, long userId, String payload) {
        String channel = channelOf(roomId, userId);
        asyncCommands.spublish(channel, payload).whenComplete((receivers, ex) -> {
            if (ex != null) {
                log.error("spublish failed, roomId={}, channel={}", roomId, channel, ex);
                return;
            }
            log.debug("spublish ok, roomId={}, channel={}, receivers={}", roomId, channel, receivers);
        });
    }

    private String channelOf(long roomId, long userId) {
        int shard = (int) (userId % 4);
        return "chat:{" + roomId + "}:" + shard;
    }

    @PreDestroy
    public void destroy() {
        if (connection != null) connection.close();
        if (client != null) client.shutdown();
    }
}

Key production notes:

Use {roomId} as a hash tag for stable routing.

Secondary sharding based on userId % 4 allows dynamic increase of shard count for hot rooms.

Gateway subscription management (Java)

package com.example.live.redis;
import io.lettuce.core.cluster.RedisClusterClient;
import io.lettuce.core.cluster.api.StatefulRedisClusterPubSubConnection;
import io.lettuce.core.cluster.pubsub.RedisClusterPubSubAdapter;
import jakarta.annotation.PostConstruct;
import jakarta.annotation.PreDestroy;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;

@Slf4j
@Component
public class GatewayShardSubscriber {
    private final Set<String> subscribedChannels = ConcurrentHashMap.newKeySet();
    private final GatewaySessionRegistry sessionRegistry;
    private RedisClusterClient client;
    private StatefulRedisClusterPubSubConnection<String, String> pubSubConnection;

    public GatewayShardSubscriber(GatewaySessionRegistry sessionRegistry) {
        this.sessionRegistry = sessionRegistry;
    }

    @PostConstruct
    public void init() {
        client = RedisClusterClient.create("redis://10.0.0.11:7001");
        pubSubConnection = client.connectPubSub();
        pubSubConnection.addListener(new RedisClusterPubSubAdapter<>() {
            @Override
            public void message(String channel, String message) {
                sessionRegistry.broadcast(channel, message);
            }
        });
    }

    public void ensureRoomSubscribed(long roomId, int shardCount) {
        List<String> channels = new ArrayList<>();
        for (int i = 0; i < shardCount; i++) {
            String channel = "chat:{" + roomId + "}:" + i;
            if (subscribedChannels.add(channel)) {
                channels.add(channel);
            }
        }
        if (!channels.isEmpty()) {
            pubSubConnection.async().ssubscribe(channels.toArray(String[]::new))
                .whenComplete((resp, ex) -> {
                    if (ex != null) {
                        channels.forEach(subscribedChannels::remove);
                        log.error("ssubscribe failed, channels={}", channels, ex);
                        return;
                    }
                    log.info("ssubscribe ok, channels={}", channels);
                });
        }
    }

    public void resubscribeAll() {
        if (subscribedChannels.isEmpty()) return;
        String[] channels = subscribedChannels.toArray(String[]::new);
        pubSubConnection.async().ssubscribe(channels)
            .whenComplete((resp, ex) -> {
                if (ex != null) {
                    log.error("resubscribe failed, count={}", channels.length, ex);
                    return;
                }
                log.warn("resubscribe ok, count={}", channels.length);
            });
    }

    @PreDestroy
    public void destroy() {
        if (pubSubConnection != null) pubSubConnection.close();
        if (client != null) client.shutdown();
    }
}

Key rule: one Redis connection per gateway node for subscriptions; avoid per‑user connections.

Sharded Pub/Sub suitability

Ideal for chat, likes, room‑wide online broadcast, low‑latency system notifications.

Not suitable for strong reliability, consumer acknowledgements, offline compensation or strict ordering – those should be handled by MQ/Stream or persistent storage.

Redis Function – governance breakthrough

Why Function matters

Function moves Lua from an ad‑hoc execution model to a centrally managed, versioned library with audit, rollback and permission control.

Typical gift‑ranking scenario

#!lua name=live_gift
local function update_gift_ranking(keys, args)
    local rank_key = keys[1]
    local score_key = keys[2]
    local effect_key = keys[3]
    local user_id = args[1]
    local gift_score = tonumber(args[2])
    local point_inc = tonumber(args[3])
    local effect_threshold = tonumber(args[4])
    local old_score = tonumber(redis.call('ZSCORE', rank_key, user_id) or '0')
    local new_score = old_score + gift_score
    redis.call('ZINCRBY', rank_key, gift_score, user_id)
    redis.call('HINCRBY', score_key, user_id, point_inc)
    local rank = redis.call('ZREVRANK', rank_key, user_id)
    local first_into_rank = (old_score == 0 and new_score > 0) and 1 or 0
    local trigger_effect = 0
    if gift_score >= effect_threshold then
        redis.call('SETEX', effect_key, 3, user_id)
        trigger_effect = 1
    end
    return { rank, new_score, first_into_rank, trigger_effect }
end
redis.register_function('update_gift_ranking', update_gift_ranking)

Java invocation

package com.example.live.gift;
import io.lettuce.core.RedisFuture;
import io.lettuce.core.cluster.api.StatefulRedisClusterConnection;
import java.util.List;
import java.util.concurrent.TimeUnit;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;

@Service
@RequiredArgsConstructor
public class GiftRankingService {
    private final StatefulRedisClusterConnection<String, String> connection;

    public GiftRankResult onGiftReceived(long roomId, long userId, int giftScore, int pointInc) {
        String rankKey = "rank:gift:" + roomId;
        String scoreKey = "score:user:" + roomId;
        String effectKey = "effect:gift:" + roomId + ":" + userId;
        RedisFuture<Object> future = connection.async().fcall(
                "update_gift_ranking",
                io.lettuce.core.ScriptOutputType.MULTI,
                new String[]{rankKey, scoreKey, effectKey},
                String.valueOf(userId),
                String.valueOf(giftScore),
                String.valueOf(pointInc),
                "1000"
        );
        try {
            @SuppressWarnings("unchecked")
            List<Object> resp = (List<Object>) future.get(300, TimeUnit.MILLISECONDS);
            long rank = ((Number) resp.get(0)).longValue();
            long newScore = ((Number) resp.get(1)).longValue();
            boolean firstIntoRank = ((Number) resp.get(2)).longValue() == 1;
            boolean triggerEffect = ((Number) resp.get(3)).longValue() == 1;
            return new GiftRankResult(rank, newScore, firstIntoRank, triggerEffect);
        } catch (Exception ex) {
            throw new IllegalStateException("fcall update_gift_ranking failed", ex);
        }
    }
}

Engineering advantages

Unified CI/CD deployment of function libraries.

Rollback by re‑loading an older function version.

Functions stay close to the data, keeping the boundary small and atomic.

Complexity review checklist

Does the script touch KEYS, SCAN or large data structures?

Is there an upper bound on execution time?

Are input parameters validated?

Are temporary objects minimized?

Is timeout and fallback strategy defined?

Release process

Local development → unit tests.

Complexity audit.

Load‑test cluster with FUNCTION LOAD.

Gray‑scale rollout.

Monitoring and verification.

Full rollout.

Rollback if needed.

Risk note

Functions still run on Redis’s single‑threaded model; heavy computation can block the server.

ACL v2 – multi‑tenant governance

Why old ACL fell short

Users had passwords but still huge permissions.

Only command categories were limited; key‑level control missing.

Temporary permission changes were manual and unaudited.

Connection pools were not bound to specific ACL users.

ACL v2 core value

Combines three dimensions – user, command and key‑pattern – to isolate services.

Sample ACL configuration

ACL SETUSER live_chat on >ChatSvc@2026 ~chat:* ~mute:* +@read +@write -@dangerous
ACL SETUSER live_gift on >GiftSvc@2026 ~rank:* ~score:* ~effect:* +@read +@write -@dangerous
ACL SETUSER live_ops_ro on >OpsRead@2026 ~rank:* ~stat:* +@read -@write -@dangerous
ACL SETUSER live_function_admin on >FuncAdmin@2026 ~* +FCALL +FUNCTION|LOAD +FUNCTION|LIST -@dangerous
ACL SETUSER default off

Interpretation:

Each service has a dedicated ACL user.

Key access is limited to the service’s prefix.

Dangerous commands are removed.

Function deployment rights are granted only to a special admin account.

Spring integration example

package com.example.live.config;
import java.util.List;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisClusterConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;

@Configuration
public class RedisAclConfig {
    @Bean
    public LettuceConnectionFactory chatRedisFactory() {
        RedisClusterConfiguration config = new RedisClusterConfiguration(
            List.of("10.0.0.11:7001", "10.0.0.12:7002", "10.0.0.13:7003")
        );
        config.setUsername("live_chat");
        config.setPassword("ChatSvc@2026");
        return new LettuceConnectionFactory(config);
    }

    @Bean
    public LettuceConnectionFactory giftRedisFactory() {
        RedisClusterConfiguration config = new RedisClusterConfiguration(
            List.of("10.0.0.11:7001", "10.0.0.12:7002", "10.0.0.13:7003")
        );
        config.setUsername("live_gift");
        config.setPassword("GiftSvc@2026");
        return new LettuceConnectionFactory(config);
    }
}

Production tips

One service → one ACL user.

Store ACL scripts in Git and generate change scripts automatically.

Audit failures: authentication errors, ACL rejections, sudden spikes after a rollout.

Combined architecture

The final architecture separates responsibilities:

Gateway handles online connections and forwards messages.

Chat service writes to chat:* via Sharded Pub/Sub.

Gift service updates rankings atomically via Function.

ACL isolates each service’s key space.

Kafka remains the “slow path” for persistence, analytics and replay.

Redis accelerates online paths but is not the sole source of truth for financial or order data.

High‑concurrency engineering upgrades

Hot‑room governance

Publish multiple shards (4‑16) for ultra‑hot rooms.

Group gateway nodes per room to reduce cross‑node sync.

Graceful degradation: sample chat, merge likes, delay non‑critical animations.

Multi‑level rate limiting (user, room, gateway).

Redis key design

chat:room:{roomId}:user:{userId}
mute:room:{roomId}
rank:gift:{roomId}
score:user:{roomId}
effect:gift:{roomId}:{userId}
stat:online:{roomId}

Key lifecycle

Short‑lived keys get TTL.

Long‑lived rankings have periodic cleanup or archiving.

Room close triggers explicit deletion.

Connection pools

Separate pools for normal read/write, Pub/Sub long‑lived connections, and admin/function deployment.

Idempotency & retry

Use businessKey + time‑window as an idempotent key inside Functions; keep strong consistency in the primary datastore.

Cluster scaling & resharding

When adding nodes, ensure hot slots are not moved unintentionally; Sharded Pub/Sub channels must be re‑subscribed automatically by gateways.

Capacity estimation dimensions

Message throughput (rooms, avg per‑room, hot peaks).

Connection counts (gateway‑Redis, gateway‑client).

Memory usage (ZSet size, Hash size, hot keys).

Network bandwidth (regular ops, Pub/Sub, replication, reshard).

Observability

Core Redis metrics

used_memory

, connected_clients, instantaneous_ops_per_sec, rejected_connections, latency.

Cluster & shard metrics

CPU, memory, network per shard; slot distribution; failover count.

Pub/Sub metrics

spublish

TPS, subscriber count per channel, hot channel traffic, subscription recovery count.

Function metrics

fcall

TPS, P95/P99 latency, error count, slow‑function count.

ACL metrics

Authentication failures, ACL rejections, high‑risk account usage.

Business‑side metrics

Chat delivery success rate, end‑to‑end latency, gift ranking update latency, effect trigger rate, ops command latency.

Typical alerts

Hot shard CPU > 75 % for 5 min.

Hot channel QPS > 3× baseline.

Function P99 > 20 ms.

ACL rejections spike after a rollout.

Gateway resubscribe count abnormal.

Rollout & rollback strategies

Sharded Pub/Sub rollout

Gateway adds support for SSUBSCRIBE and SPUBLISH.

Enable double‑write to old and new channels.

Monitor latency and consistency.

Gray‑scale hot rooms to shard channel.

Full switch‑over.

Disable old PUBLISH/SUBSCRIBE path.

Function rollout

Load function library in pre‑prod.

Switch a low‑traffic service to FCALL.

Keep old Lua path for side‑by‑side comparison.

Full switch after metrics stabilize.

Remove old EVALSHA usage.

ACL rollout

Map current command/key usage.

Run audit mode (read‑only) scripts.

Bind each service to a dedicated ACL user.

Observe rejection logs.

Gradually expand scope.

Disable default user.

Rollback

Each module has an independent rollback plan: revert Pub/Sub channel, reload previous function version, or restore previous ACL set. All steps must be rehearsed in a disaster‑recovery drill.

Benchmark results

Chat P99 latency: 200 ms → 32 ms.

Cluster broadcast traffic: 1.0× baseline → 0.34× baseline.

Ranking network round‑trips: 4 times → 1 time.

Hot‑room CPU peak: 92 % → 61 %.

Redis accidental faults: occurred → zero.

Failover subscription recovery: manual → auto < 3 s.

Common pitfalls

Sharded Pub/Sub is not a universal replacement for all message middleware – it only fits low‑latency broadcast.

Function should host small, atomic logic; do not move complex application code into Redis.

ACL alone does not guarantee safety – it must be combined with per‑service users and key patterns.

Focusing only on Redis metrics while ignoring user‑experience indicators leads to hidden problems.

Scaling nodes alone cannot solve hot‑room bottlenecks; routing, sharding and graceful degradation are essential.

Actionable recommendations for teams

Audit existing Pub/Sub usage; if real‑time broadcast dominates, prioritize Sharded Pub/Sub.

Identify scattered Lua scripts; migrate high‑frequency ones to Functions.

Separate service identities, enforce key‑prefix conventions, and apply ACL v2.

Follow the 10‑step checklist (channel design, secondary sharding, connection pool separation, monitoring, rollout, rollback, etc.) to turn Redis 7 features into a production‑ready foundation.

Appendix – minimal practice checklist

Catalog all Redis broadcast scenarios and classify real‑time vs reliable.

Define a unified channel naming scheme with secondary sharding hooks.

Implement gateway SSUBSCRIBE with auto‑resubscribe and double‑write switch.

Extract high‑frequency Lua into Functions and store them in a version‑controlled repo.

Establish Function complexity review and performance testing.

Split ACL accounts per service and enforce key‑pattern isolation.

Disable the default user and collapse dangerous commands.

Separate Redis connections into read/write, Pub/Sub, and admin pools.

Set up joint Redis‑and‑business alerting.

Document full rollout, gray‑scale, rollback, and failure‑drill procedures.