Build a Distributed WebSocket Cluster with Spring Cloud & Consistent Hashing

When working on a project that required communication between multiple users, I encountered issues with WebSocket handshake requests and sharing WebSocket sessions across a cluster.

After several days of research, I summarized several methods for implementing a distributed WebSocket cluster, experimenting with Zuul and Spring Cloud Gateway, and compiled this article to help others and share ideas.

System Architecture Diagram

In my implementation, each application server handles both HTTP and WebSocket requests, although the chat model could be separated into its own module. From a distributed perspective, both approaches are similar, but combining HTTP and WebSocket in a single service is more convenient.

Technology stack involved:

Eureka service discovery and registration

Redis session sharing

Redis message subscription

Spring Boot

Zuul gateway

Spring Cloud Gateway

Spring WebSocket for long connections

Ribbon load balancing

Netty multi‑protocol NIO network framework

Consistent Hash algorithm

WebSocketSession vs HttpSession:

In Spring's WebSocket integration, each WebSocket connection has a corresponding WebSocketSession. After establishing a connection, you can communicate with the client like this:

protected void handleTextMessage(WebSocketSession session, TextMessage message) {
    System.out.println("Server received message: " + message);
    // send message to client
    session.sendMessage(new TextMessage("message"));
}

WebSocket sessions cannot be serialized to Redis, so they cannot be shared across the cluster like HttpSession. Attempting to cache only key information and reconstruct sessions is impractical.

HttpSession sharing is straightforward with dependencies: spring-session-data-redis and spring-boot-starter-redis.

Netty vs Spring WebSocket:

Using Netty for WebSocket

Netty does not have a WebSocket session concept; instead it uses channels. A channel represents each client connection, and handlers process messages.

private static final ChannelGroup GROUP = new DefaultChannelGroup(ImmediateEventExecutor.INSTANCE);
@Override
protected void channelRead0(ChannelHandlerContext ctx, TextWebSocketFrame msg) throws Exception {
    System.out.println("Server received from " + ctx.channel().id() + ": " + msg.text());
    GROUP.writeAndFlush(msg.retain());
}

Pros of Netty include high concurrency, but drawbacks are integration difficulty with other services, duplicated business logic, and lack of built‑in REST support.

Using Spring WebSocket

Spring WebSocket is well integrated with Spring Boot, making development simple.

Step 1: Add dependency

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-websocket</artifactId>
</dependency>

Step 2: Add configuration class

@Configuration
public class WebSocketConfig implements WebSocketConfigurer {
    @Override
    public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
        registry.addHandler(myHandler(), "/").setAllowedOrigins("*");
    }
    @Bean
    public WebSocketHandler myHandler() {
        return new MessageHandler();
    }
}

Step 3: Implement message handler

@Component
public class MessageHandler extends TextWebSocketHandler {
    private List<WebSocketSession> clients = new ArrayList<>();
    @Override
    public void afterConnectionEstablished(WebSocketSession session) {
        clients.add(session);
        System.out.println("uri:" + session.getUri());
        System.out.println("Connection established:" + session.getId());
        System.out.println("Current sessions:" + clients.size());
    }
    @Override
    public void afterConnectionClosed(WebSocketSession session, CloseStatus status) {
        clients.remove(session);
        System.out.println("Connection closed:" + session.getId());
    }
    @Override
    protected void handleTextMessage(WebSocketSession session, TextMessage message) {
        String payload = message.getPayload();
        Map<String, String> map = JSONObject.parseObject(payload, HashMap.class);
        System.out.println("Received data" + map);
        clients.forEach(s -> {
            try {
                System.out.println("Sending to:" + session.getId());
                s.sendMessage(new TextMessage("Server echo:" + payload));
            } catch (Exception e) {
                e.printStackTrace();
            }
        });
    }
}

Using Spring WebSocket aligns the service with the Spring Cloud ecosystem, so I chose this approach.

Architecture: each application server provides both RESTful APIs and WebSocket services, avoiding a separate WebSocket module to reduce extra I/O layers.

Transition from Zuul to Spring Cloud Gateway

Zuul 1.0 does not support WebSocket forwarding; Zuul 2.0 does but is not integrated with Spring Boot. Therefore, migration to Spring Cloud Gateway is necessary.

Gateway SSL and dynamic routing require specific YAML configuration (omitted for brevity). Additionally, an HTTPS‑to‑HTTP filter is needed to avoid "not an SSL/TLS record" errors:

@Component
public class HttpsToHttpFilter implements GlobalFilter, Ordered {
    private static final int HTTPS_TO_HTTP_FILTER_ORDER = 10099;
    @Override
    public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
        URI originalUri = exchange.getRequest().getURI();
        ServerHttpRequest request = exchange.getRequest();
        ServerHttpRequest.Builder mutate = request.mutate();
        String forwardedUri = request.getURI().toString();
        if (forwardedUri != null && forwardedUri.startsWith("https")) {
            try {
                URI mutatedUri = new URI("http",
                    originalUri.getUserInfo(),
                    originalUri.getHost(),
                    originalUri.getPort(),
                    originalUri.getPath(),
                    originalUri.getQuery(),
                    originalUri.getFragment());
                mutate.uri(mutatedUri);
            } catch (Exception e) {
                throw new IllegalStateException(e.getMessage(), e);
            }
        }
        ServerHttpRequest build = mutate.build();
        ServerWebExchange webExchange = exchange.mutate().request(build).build();
        return chain.filter(webExchange);
    }
    @Override
    public int getOrder() {
        return HTTPS_TO_HTTP_FILTER_ORDER;
    }
}

With the gateway configured, both HTTPS and WSS requests can be forwarded, and the next step is to enable session communication across the cluster.

Session Broadcast Solution

This is the simplest WebSocket cluster communication method. A teacher sends a broadcast message to students; the gateway forwards the request to all cluster nodes, each node checks for associated sessions and sends the message if present. This approach wastes computation when no matching sessions exist but works for low concurrency.

Example of obtaining all service instances via Eureka:

@Resource
private EurekaClient eurekaClient;
Application app = eurekaClient.getApplication("service-name");
InstanceInfo instanceInfo = app.getInstances().get(0);
System.out.println("ip address: " + instanceInfo.getIPAddr());

Maintain a mapping table of user IDs to sessions; add entries when sessions are created and remove them when closed.

Consistent Hashing Solution (Key Point)

This method is considered the most elegant. It requires understanding consistent hashing. The hash ring maps user IDs to specific cluster nodes, ensuring that messages are routed to the server holding the corresponding WebSocket session.

Challenges include handling node down/up events and keeping the hash ring synchronized. When a node goes down, its virtual nodes are removed from the ring; when a node comes up, virtual nodes are added, and affected sessions must reconnect.

Two possible placements for the hash ring:

Gateway maintains the ring locally – not recommended due to coordination complexity.

Eureka creates the ring and stores it in Redis; gateways subscribe to Redis updates to stay consistent.

Load balancing: Spring Cloud Gateway (or Zuul) uses Ribbon. By customizing Ribbon's load‑balancing rule to hash the user ID, the request can be directed to the appropriate node based on the hash ring.

Because Ribbon in the Finchley release lacks a proper key parameter, a workaround is to have the client first make an HTTP request containing the user ID, receive the target IP from the gateway, and then establish the WebSocket connection to that IP.

Conclusion

The article presented two WebSocket cluster solutions—simple session broadcasting and consistent hashing—each suited to different scenarios. It also discussed the limitations of current Ribbon implementations and provided practical code snippets and configuration examples for building a robust distributed WebSocket system.