Scaling Real-Time WebSocket Push Across Distributed Instances with Redis

In distributed systems, handling real-time WebSocket message push across multiple instances requires coordination. This article describes a solution using Redis as a coordination middleware, defining Client and ClientManager components in Go, and outlines the message flow from producer to end‑users.

Basic Principle

To meet the requirement, Redis is used as a coordination middleware to store user information, generate unique identifiers for connections, and keep pod addresses. The message‑producing instance subscribes to Redis to obtain these identifiers and notifies the appropriate message instance, which then pushes the message to the client via WebSocket.

Server Implementation

Client

The Client component manages a single user connection after it is established with a message service instance.

type Client struct {
    UUID   string
    UserID string
    Socket *websocket.Conn
    Send   chan []byte
}

Fields:

UUID – unique identifier for the connection.

UserID – user identifier.

Socket – the WebSocket connection object.

Send – channel for outgoing message data.

Two methods are provided:

Read method

func (c *Client) Read(close, renewal chan *Client) {
    defer func() { close <- c }()
    for {
        _, message, err := c.Socket.ReadMessage()
        if err != nil {
            break
        }
        // ... message logic ...
    }
}

Write method

func (c *Client) Write(close chan *Client) {
    for {
        select {
        case message, ok := <-c.Send:
            if !ok {
                return
            }
            c.Socket.WriteMessage(websocket.TextMessage, message)
        case <-c.Ctx.Done():
            return
        }
    }
}

ClientManager

The ClientManager acts as a connection pool, managing all client connections and providing registration, deregistration, and renewal functions.

type ClientManager struct {
    sync.RWMutex
    Clients    map[string]*Client
    Register   chan *Client
    Unregister chan *Client
    Renewal    chan *Client
}

Key fields:

Clients – map storing all active Client objects.

Register – channel for new connections.

Unregister – channel for removing connections.

Renewal – channel for Redis key renewal.

The Start method listens on these channels and performs the corresponding actions.

func (manager *ClientManager) Start(ctx context.Context) {
    for {
        select {
        case conn := <-manager.Register:
            manager.Lock()
            manager.Clients[conn.UUID] = conn
            manager.Unlock()
            // register in Redis
        case conn := <-manager.Unregister:
            // unregister from Redis and clean up
            conn.Socket.Close()
            close(conn.Send)
            delete(manager.Clients, conn.UUID)
        case conn := <-manager.Renewal:
            // renew Redis key
        }
    }
}

Message Push

When a message service instance generates a user message, the push steps are:

Read the user’s connection identifier and pod address from Redis.

Send a request to the target message service instance identified by the pod address.

The target instance receives the request and locates the corresponding Client.

The server handling logic includes:

Write to Client

func (manager *ClientManager) Write(message *Message) error {
    manager.RLock()
    client, ok := manager.Clients[message.Recipient]
    manager.RUnlock()
    if !ok {
        return errors.New("client miss [" + message.Recipient + "]")
    }
    return client.SendOut(message)
}

Client SendOut

func (c *Client) SendOut(message *Message) error {
    content, err := json.Marshal(message.Content)
    if err != nil {
        return err
    }
    c.Send <- content
    return nil
}

The Client’s Write method continuously reads from the Send channel and forwards the data to the terminal via the WebSocket connection.

Summary

The core idea for WebSocket message pushing in a distributed environment is to store user connection identifiers and pod addresses in Redis, allowing any message‑producing instance to locate the correct instance that holds the active client connection and deliver the message through WebSocket. This approach ensures real‑time delivery across multiple service instances.