Spring Boot + Netty MQTT Gateway for Million Connections & Millisecond Push

Why "connected" is not the same as "online"

Many teams start with a simple MQTT broker, but when traffic reaches millions of long‑lived connections the real challenges appear: mapping ClientId->Channel blows up, authentication blocks Netty I/O threads, upstream messages reach Kafka while device state is not unified, cluster scaling loses the target node for downstream messages, and Kubernetes rolling updates kill pods causing massive reconnections.

The core difficulty is connection‑lifecycle governance, not just message receipt.

What the system must solve

A production‑grade MQTT gateway must satisfy four categories of requirements:

2.1 Connection handling

TCP/TLS listening, MQTT codec, CONNECT authentication, heartbeat, duplicate‑connection kicking, session persistence and protocol compatibility.

2.2 Message routing

Upstream telemetry, downstream commands, broadcast notifications and acknowledgments must be routed to distinct event channels, supporting multi‑tenant, multi‑product and multi‑device type coexistence.

2.3 State convergence

Online status, last‑active time, subscription relations, device shadow, command execution state and offline pending messages form the state plane. Without this plane the platform cannot know whether a device actually received or executed a command.

2.4 Governance & operations

Rate limiting, back‑pressure, circuit breaking, logging, metrics, tracing, gray releases, fault isolation, capacity planning and loss‑less scaling are essential for production readiness.

Overall production‑ready architecture

The gateway is split into four planes: Access, Event, State and Governance. The diagram (textual) shows the device side (MQTT over TCP/TLS) feeding the Netty gateway (access plane), Kafka/Stream (event plane), Redis + DB (state plane) and Prometheus/Tracing/Config/HPA (governance plane).

3.1 Access plane

Listen on TCP/TLS ports.

Decode MQTT packets.

Handle CONNECT, SUBSCRIBE, PUBLISH, PINGREQ, DISCONNECT.

Maintain local active connection and subscription indexes.

Offload heavy logic asynchronously to downstream planes.

Principle: keep the I/O thread light; never perform slow operations there.

3.2 Event plane

Upstream telemetry is written to Kafka.

Downstream commands go to a command topic.

Internal routing messages go to a gateway‑route topic.

Dead‑letter, compensation and audit messages have dedicated channels.

This decouples inbound traffic from business processing, preventing the gateway from being overwhelmed during spikes.

3.3 State plane

Redis stores online sessions, node affinity, recent heartbeat timestamps and short‑term idempotent keys.

Database stores device shadow, command state machine, offline messages and audit logs.

In‑memory read‑only indexes such as clientId->Channel and topic->local subscribers accelerate look‑ups.

Redis is used only for high‑frequency session data; the database holds the ultimate facts.

3.4 Governance plane

Metrics: connection count, auth latency, message throughput, off‑heap memory, event‑loop delay.

Downstream success rate, command backlog, offline compensation count, reconnection storm alerts.

Hot‑config updates, rate‑limit adjustments, gray‑node traffic shedding, abnormal node isolation.

Without this plane “million long connections” is just a slogan.

Why choose Spring Boot + Netty instead of a pure broker

4.1 Scenarios where a self‑built gateway makes sense

Deep coupling of device authentication with corporate identity.

Fine‑grained routing by tenant, product, region or device type.

Downstream commands need to be linked with orders, jobs, work orders or rule‑engine state.

Need to bring the connection layer into Spring’s configuration, service‑discovery, monitoring and release pipeline.

Control over protocol extensions, custom headers, link tags and audit logic.

4.2 When a commercial broker is preferable

No deep custom CONNECT auth required.

Downstream command volume is low, no complex state machine.

Team lacks long‑term Netty, protocol compatibility and high‑concurrency tuning expertise.

Speed of delivery outweighs unified architectural control.

Real‑world business flow example

A campus energy platform with air‑conditioners, lights and power meters illustrates the end‑to‑end flow:

Device connects via TLS, sending clientId, tenant ID and signature in CONNECT.

Gateway authenticates, registers the session in Redis and returns CONNACK.

Device periodically publishes telemetry (temperature, current, power, mode).

Gateway asynchronously writes the upstream message to Kafka; rule service decides whether to issue an energy‑saving command.

Command service creates a command record, writes to the command state machine and Outbox, then routes the command to the appropriate gateway node.

Gateway finds the target device connection and pushes the command; device executes and sends an acknowledgment.

Platform updates the command state from CREATED → SENT → ACKED → SUCCEEDED or TIMEOUT.

If the device is offline, the command enters an offline‑pending queue and is re‑delivered after reconnection.

This demonstrates that “MQTT push” is not a single writeAndFlush call but a closed‑loop across connections, nodes and state planes.

Key protocol and connection decisions

6.1 QoS selection

Telemetry usually uses QoS 1.

High‑frequency low‑value events can use QoS 0.

QoS 2 is reserved for extremely duplicate‑sensitive commands and still requires idempotence at the business layer.

QoS 2 incurs extra server‑side state, retransmission windows and timeout cleanup, increasing memory pressure under high connection counts.

6.2 KeepAlive handling

Treat KeepAlive as a reference, not a hard timeout.

Set server idle‑kick threshold to 1.5‑2 × KeepAlive.

Adjust per‑device based on recent RTT, device type and network conditions.

6.3 Clean Session / Session Expiry

If offline devices must still receive critical commands, the system must explicitly retain session state, define its expiry and decide which topics allow replay; otherwise downstream reliability is merely a slogan.

6.4 Topic design

up/{tenantId}/{productKey}/{deviceId}/{messageType}
 down/{tenantId}/{productKey}/{deviceId}/{commandType}
 reply/{tenantId}/{productKey}/{deviceId}/{commandId}
 shadow/{tenantId}/{productKey}/{deviceId}/reported
 shadow/{tenantId}/{productKey}/{deviceId}/desired

This hierarchical scheme enables clear routing, multi‑tenant isolation, aggregated monitoring, natural command‑reply correlation and easier bridging or replay later.

Netty threading model

7.1 Design principles

I/O threads only handle connection and protocol parsing.

Slow operations must be offloaded to business thread pools.

Authentication, DB lookups or remote calls must never block a ChannelHandler synchronously.

Responses must be controllable to avoid unbounded buffering per connection.

7.2 Recommended pipeline

MqttDecoder
-> MqttEncoder
-> ConnectionQuotaHandler
-> IdleStateHandler
-> ProtocolGuardHandler
-> AuthDispatchHandler
-> PublishIngressHandler
-> SubscribeHandler
-> ExceptionAndMetricsHandler

Key handlers: ConnectionQuotaHandler: basic rate‑limiting (IP connection caps, burst limits). ProtocolGuardHandler: validates packet size, illegal topics, reserved bits and rejects malformed frames. AuthDispatchHandler: dispatches authentication to an async executor, never blocks the I/O thread. PublishIngressHandler: quickly copies payload, performs lightweight validation and hands the envelope to the event service.

7.3 Value of Spring Boot

Manages configuration, thread pools, monitoring, metric export and lifecycle.

Hosts command, state and routing beans.

Handles graceful shutdown on SIGTERM.

Integrates with Prometheus, OpenTelemetry, Nacos/Apollo, etc.

Netty provides high‑performance networking; Spring Boot provides engineering, governance and observability.

Production‑grade code skeleton

8.1 Gateway start‑up and graceful stop

@Component
public class MqttGatewayServer implements SmartLifecycle {
    private final GatewayChannelInitializer channelInitializer;
    private final GatewayDrainManager drainManager;
    private final GatewayProperties properties;
    private EventLoopGroup bossGroup;
    private EventLoopGroup workerGroup;
    private Channel serverChannel;
    private volatile boolean running;

    public MqttGatewayServer(GatewayChannelInitializer channelInitializer,
                             GatewayDrainManager drainManager,
                             GatewayProperties properties) {
        this.channelInitializer = channelInitializer;
        this.drainManager = drainManager;
        this.properties = properties;
    }

    @Override
    public void start() {
        bossGroup = new NioEventLoopGroup(properties.getBossThreads());
        workerGroup = new NioEventLoopGroup(properties.getWorkerThreads());
        ServerBootstrap bootstrap = new ServerBootstrap();
        bootstrap.group(bossGroup, workerGroup)
                 .channel(NioServerSocketChannel.class)
                 .option(ChannelOption.SO_BACKLOG, properties.getSoBacklog())
                 .childOption(ChannelOption.TCP_NODELAY, true)
                 .childOption(ChannelOption.SO_KEEPALIVE, true)
                 .childOption(ChannelOption.WRITE_BUFFER_WATER_MARK,
                              new WriteBufferWaterMark(32 * 1024, 256 * 1024))
                 .childHandler(channelInitializer);
        try {
            serverChannel = bootstrap.bind(properties.getPort()).sync().channel();
            running = true;
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new IllegalStateException("Failed to start mqtt gateway", e);
        }
    }

    @Override
    public void stop() {
        drainManager.enterDraining();
        if (serverChannel != null) {
            serverChannel.close();
        }
        workerGroup.shutdownGracefully();
        bossGroup.shutdownGracefully();
        running = false;
    }

    @Override
    public boolean isRunning() { return running; }
}

Three production‑level details matter more than merely listening on a port:

Use WRITE_BUFFER_WATER_MARK to bound per‑connection outbound buffers.

During shutdown first enter a draining state, then stop accepting new traffic, finally wait for existing connections to migrate.

All tunable parameters (thread counts, ports, watermarks) are externalized in configuration.

8.2 Non‑blocking CONNECT authentication

@ChannelHandler.Sharable
public class AuthDispatchHandler extends SimpleChannelInboundHandler<MqttConnectMessage> {
    private final DeviceAuthService deviceAuthService;
    private final GatewaySessionRegistry sessionRegistry;
    private final Executor authExecutor;

    public AuthDispatchHandler(DeviceAuthService deviceAuthService,
                               GatewaySessionRegistry sessionRegistry,
                               @Qualifier("authExecutor") Executor authExecutor) {
        this.deviceAuthService = deviceAuthService;
        this.sessionRegistry = sessionRegistry;
        this.authExecutor = authExecutor;
    }

    @Override
    protected void channelRead0(ChannelHandlerContext ctx, MqttConnectMessage msg) {
        String clientId = msg.payload().clientIdentifier();
        byte[] passwordBytes = msg.payload().passwordInBytes();
        String username = msg.payload().userName();
        String password = passwordBytes == null ? null : new String(passwordBytes, StandardCharsets.UTF_8);
        CompletableFuture.supplyAsync(() ->
                deviceAuthService.authenticate(clientId, username, password), authExecutor)
            .whenComplete((result, error) -> {
                if (error != null || !result.isSuccess()) {
                    writeConnAckAndClose(ctx, MqttConnectReturnCode.CONNECTION_REFUSED_BAD_USER_NAME_OR_PASSWORD);
                    return;
                }
                try {
                    sessionRegistry.register(result.devicePrincipal(), ctx.channel(), msg.variableHeader().isCleanSession());
                    ctx.channel().attr(SessionAttributes.CLIENT_ID).set(clientId);
                    ctx.writeAndFlush(MqttMessageBuilders.connAck()
                        .returnCode(MqttConnectReturnCode.CONNECTION_ACCEPTED)
                        .sessionPresent(false)
                        .build());
                } catch (DuplicateClientException ex) {
                    writeConnAckAndClose(ctx, MqttConnectReturnCode.CONNECTION_REFUSED_IDENTIFIER_REJECTED);
                }
            });
    }

    private void writeConnAckAndClose(ChannelHandlerContext ctx, MqttConnectReturnCode code) {
        ctx.writeAndFlush(MqttMessageBuilders.connAck().returnCode(code).build())
            .addListener(ChannelFutureListener.CLOSE);
    }
}

The handler emphasizes asynchronous auth plus session registration; real deployments also check device freeze status, tenant quota, node admission, client‑certificate fingerprint and token revocation.

8.3 Upstream message ingestion

public class PublishIngressHandler extends SimpleChannelInboundHandler<MqttPublishMessage> {
    private final GatewayIngressService ingressService;
    public PublishIngressHandler(GatewayIngressService ingressService) { this.ingressService = ingressService; }
    @Override
    protected void channelRead0(ChannelHandlerContext ctx, MqttPublishMessage msg) {
        String clientId = ctx.channel().attr(SessionAttributes.CLIENT_ID).get();
        String topic = msg.variableHeader().topicName();
        byte[] payload = new byte[msg.payload().readableBytes()];
        msg.payload().getBytes(msg.payload().readerIndex(), payload);
        ingressService.accept(new IngressEnvelope(clientId, topic, payload,
                msg.fixedHeader().qosLevel().value(), msg.variableHeader().packetId()));
        if (msg.fixedHeader().qosLevel() == MqttQoS.AT_LEAST_ONCE) {
            ctx.writeAndFlush(MqttMessageBuilders.pubAck()
                .packetId(msg.variableHeader().packetId())
                .build());
        }
    }
}

Fast I/O thread capture followed by async hand‑off to ingressService prevents blocking on Kafka, JSON parsing or rule evaluation.

8.4 Business ingress service – idempotence and Kafka publishing

@Service
public class GatewayIngressService {
    private final RedisTemplate<String, String> redisTemplate;
    private final KafkaTemplate<String, byte[]> kafkaTemplate;
    private final TopicRouter topicRouter;
    public void accept(IngressEnvelope envelope) {
        String idempotentKey = "mqtt:dup:" + envelope.clientId() + ":" + envelope.packetId();
        Boolean firstSeen = redisTemplate.opsForValue()
            .setIfAbsent(idempotentKey, "1", Duration.ofMinutes(10));
        if (Boolean.FALSE.equals(firstSeen) && envelope.qos() > 0) { return; }
        String kafkaTopic = topicRouter.toUpstreamTopic(envelope.topic());
        ProducerRecord<String, byte[]> record = new ProducerRecord<>(kafkaTopic,
                envelope.clientId(), envelope.payload());
        record.headers().add("mqtt-topic", envelope.topic().getBytes(StandardCharsets.UTF_8));
        record.headers().add("source-client-id", envelope.clientId().getBytes(StandardCharsets.UTF_8));
        kafkaTemplate.send(record).whenComplete((result, ex) -> {
            if (ex != null) { publishDeadLetter(envelope, ex); }
        });
    }
}

The service guarantees at‑least‑once delivery by short‑term deduplication in Redis and defers failures to a dead‑letter flow.

Clustered downstream delivery challenges

Naïve single‑node write channel.writeAndFlush(message) fails in a cluster because the command service does not know which node holds the device’s channel.

A robust approach:

On successful CONNECT, write clientId->gatewayNodeId to Redis.

Command service creates a command record with state CREATED.

Within the same transaction write an DeviceCommandCreated outbox event.

Outbox publisher pushes the event to a Kafka command topic.

Routing consumer looks up the node ID in Redis and publishes the command to a node‑specific topic.

Target gateway consumes the command, sends it downstream, and reports SENT / FAILED / TIMEOUT.

This separates “state facts” from “connection facts”, allowing business logic to operate on commands without touching raw sockets.

9.1 Downstream routing flow

Device → Target gateway node → Redis → Kafka → Command service → Outbox → Node‑specific topic → Gateway → MQTT PUBLISH → Device → Ack → Command state update.

9.2 Command state machine

CREATED

ROUTING

SENT

ACKED

SUCCEEDED

FAILED

TIMEOUT

CANCELLED

Without this explicit state machine the business can only guess success from logs.

Device shadow, offline messages and compensation

Three typical production questions:

Where to store the desired state when a device is offline?

Which commands need re‑delivery after reconnection?

When multiple commands arrive, which one wins?

All belong to state convergence.

10.1 Device shadow

Maintain two records per device: desired (platform intent) and reported (last confirmed device state). The platform writes desired first; only after receiving a device acknowledgment does it update reported. This makes offline intent visible.

10.2 Offline compensation

Simple in‑memory queues lose data on node restart, cannot migrate across nodes and lack auditability. The recommended pattern stores command facts in a database, short‑term pending indexes in Redis, and lets the state service scan and re‑route pending commands after reconnection.

10.3 Compensation boundaries

Telemetry is usually not replayed.

Critical control commands are replayed with version or idempotent keys.

Transient notifications are not replayed.

Clear business rules prevent the “absolute reliability” myth that inflates system complexity.

High‑concurrency bottlenecks

11.1 Slow operations dominate

CONNECT auth that calls remote services.

PUBLISH handling that synchronously serialises or writes to DB.

Downstream routing that depends on hot Redis keys.

Authentication storms caused by mass reconnections.

Capacity planning must consider new‑connection rate, auth request rate, peak message volume and downstream command peaks, not just total connections.

11.2 Explicit back‑pressure

Monitor write‑buffer watermarks to detect non‑writable channels.

Set per‑device pending‑send queue limits.

When thresholds are exceeded, drop low‑value messages or close abnormal connections.

Batch broadcast deliveries instead of flooding all devices at once.

11.3 Authentication storms

Cache recent auth results briefly.

Apply timeout‑based circuit breakers to the auth service.

Rate‑limit connections per IP or per tenant.

During traffic‑shedding, reject new connections while preserving existing ones for migration.

11.4 Kafka as a back‑pressure source

Distinguish core topics from ordinary ones.

Assign different timeouts and retry policies per message type.

Allow low‑value messages to be dropped under extreme load.

Monitor inbound queue lengths and trigger automatic protection.

Kubernetes long‑connection governance

12.1 Why naïve rolling update breaks

Massive simultaneous disconnections.

Reconnections overload remaining pods.

In‑flight commands are interrupted, leaving business state dangling.

Session affinity in Redis briefly becomes inconsistent.

This creates a cascade: deployment → traffic jitter → reconnection storm → auth and Redis overload.

12.2 Safe drain‑first shutdown

Node enters draining state, stops accepting new connections.

Remove the node from service‑discovery or load‑balancer (摘流).

Signal existing connections with a migration window; let naturally disconnecting devices migrate first.

Give critical business a short window to finish pending downstream commands.

After timeout, close remaining connections.

12.3 Example Pod spec (illustrative)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mqtt-gateway
spec:
  replicas: 6
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: gateway
          image: example/mqtt-gateway:1.0.0
          ports:
            - containerPort: 1883
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "curl -s http://127.0.0.1:8080/actuator/gateway/drain && sleep 45"]
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080

The key is that the application implements a true drain semantics, not just a sleep.

Observability for rapid troubleshooting

When a fault occurs, operators should look at four metric families simultaneously.

13.1 Connection metrics

Current online connections.

Connections per second (established / closed).

CONNECT auth success rate.

Duplicate‑connection (top‑kick) count.

Heartbeat timeout disconnects.

13.2 Message metrics

Upstream QPS / downstream QPS.

Kafka send latency.

Downstream routing success rate.

Command‑ack timeout count.

Offline compensation queue backlog.

13.3 Resource metrics

EventLoop pending task count.

JVM heap, off‑heap and direct memory usage.

File‑descriptor utilization.

Socket write‑buffer watermarks.

Redis / Kafka client timeout rates.

13.4 Business‑state metrics

Distribution of command‑state machine statuses.

Device‑shadow convergence lag.

Per‑tenant connection share.

Hot topics per product.

Correlating these signals reveals patterns such as:

Connection surge + auth timeout + Redis latency ⇒ reconnection storm.

Downstream failure rise + non‑writable channels + command backlog ⇒ device consumption bottleneck.

Heartbeat loss + regional network anomalies ⇒ network issue, not server crash.

When to adopt a self‑built gateway vs a commercial broker

14.1 Ideal moments for self‑development

Device count exceeds what a single broker node can handle.

Need to integrate with internal authentication, audit and observability systems.

Downstream commands are core business flows.

Team can maintain Netty, Redis, Kafka and cloud‑native release pipelines long‑term.

14.2 Situations where a broker is more economical

Device fleet is still small.

Business logic is not yet stable.

Rule engine, device shadow and command compensation are not fully defined.

14.3 Incremental evolution path

Start with a single‑node gateway to validate protocol and auth.

Add Kafka to split out the event plane.

Introduce Redis for session affinity and cross‑node routing, enabling horizontal scaling.

Implement command state machine, device shadow and Outbox for a complete business loop.

Add drain, gray release, auto‑scaling and full observability to reach stable production.

Each step adds clear value and solves a specific problem.

Final takeaway

Spring Boot + Netty can deliver a high‑performance MQTT gateway, but success hinges on three layers:

Connection layer: stable acceptance, protocol parsing, heartbeat and lifecycle management.

State layer: unified online session, offline compensation, device shadow and command acknowledgment loops.

Governance layer: explicit rate‑limits, back‑pressure, graceful Kubernetes roll‑outs and loss‑less scaling.

Only when all these capabilities coexist does the system move from a runnable demo to a production‑grade ingress infrastructure.