Beyond HTTP: Building a Custom TCP Protocol with Spring Boot and Netty from Scratch

Why hand‑written sockets are problematic

Multithreading model becomes complex.

Resource management under high concurrency is difficult.

Heartbeat, timeout, half‑packet / sticky‑packet handling must be implemented manually.

Resulting code is hard to maintain and less stable.

Netty as a better alternative

Netty provides an event‑driven, asynchronous, non‑blocking architecture with a mature codec system and strong extensibility, allowing developers to focus on business protocol logic instead of low‑level I/O.

Netty core threading model

BossGroup : listens on ports, accepts incoming connections, and assigns them to workers.

WorkerGroup : performs network read/write and invokes ChannelHandler for business processing.

Environment preparation

Maven dependencies (Java 8, Netty 4.1.65.Final, optional Spring Boot Web 2.7.18):

<properties>
    <maven.compiler.source>8</maven.compiler.source>
    <maven.compiler.target>8</maven.compiler.target>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<dependencies>
    <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty-all</artifactId>
        <version>4.1.65.Final</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
        <version>2.7.18</version>
    </dependency>
</dependencies>

Netty service construction

Create Boss and Worker groups

// Boss: accept connections
EventLoopGroup bossGroup = new NioEventLoopGroup();

// Worker: handle read/write
EventLoopGroup workerGroup = new NioEventLoopGroup();

Build ServerBootstrap

ServerBootstrap bootstrap = new ServerBootstrap();
bootstrap.group(bossGroup, workerGroup)
         .channel(NioServerSocketChannel.class);

Initialize channel pipeline

bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
    @Override
    protected void initChannel(SocketChannel ch) {
        // Decoder (top of the stack)
        ch.pipeline().addLast(new DeviceMessageDecoder(49));
        // Read timeout detection (10 minutes)
        ch.pipeline().addLast("idleStateHandler", new IdleStateHandler(600, 0, 0));
        // Business handler
        ch.pipeline().addLast("device-handler", new DeviceServerHandler());
    }
});

Custom protocol decoder design

Protocol characteristics:

Fixed length of 49 bytes .

Start marker FB 90.

Binary payload ultimately processed as a hexadecimal string.

Implementation extends ByteToMessageDecoder:

package com.icoderoad.platform.decoder;

import io.netty.buffer.ByteBuf;
import io.netty.channel.ChannelHandlerContext;
import io.netty.handler.codec.ByteToMessageDecoder;
import io.netty.util.ByteProcessor;
import java.util.List;

/**
 * Custom device message decoder for a fixed‑length 16‑hex protocol.
 */
public class DeviceMessageDecoder extends ByteToMessageDecoder {
    private final int frameLength;

    public DeviceMessageDecoder(int frameLength) {
        this.frameLength = frameLength;
    }

    @Override
    protected void decode(ChannelHandlerContext ctx, ByteBuf buf, List<Object> out) {
        int fbIndex = buf.forEachByte(new ByteProcessor.IndexOfProcessor((byte)0xFB));
        int nextIndex = buf.forEachByte(new ByteProcessor.IndexOfProcessor((byte)0x90));
        if (fbIndex != -1 && nextIndex == fbIndex + 1) {
            buf.readerIndex(fbIndex);
            if (buf.readableBytes() >= frameLength) {
                ByteBuf slice = buf.readRetainedSlice(frameLength);
                byte[] data = new byte[frameLength];
                slice.readBytes(data);
                out.add(bytesToHex(data));
                slice.release();
            }
        }
    }

    private String bytesToHex(byte[] bytes) {
        StringBuilder sb = new StringBuilder();
        for (byte b : bytes) {
            String hex = Integer.toHexString(b & 0xFF);
            sb.append(hex.length() == 1 ? "0" : "").append(hex.toUpperCase());
        }
        return sb.toString();
    }
}

Business event handler

package com.icoderoad.platform.handler;

import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.SimpleChannelInboundHandler;
import io.netty.handler.timeout.IdleStateEvent;
import io.netty.buffer.Unpooled;
import java.nio.charset.StandardCharsets;

public class DeviceServerHandler extends SimpleChannelInboundHandler<Object> {
    @Override
    protected void channelRead0(ChannelHandlerContext ctx, Object msg) {
        String hexData = (String) msg;
        // Business logic: parse, store, respond
    }

    @Override
    public void userEventTriggered(ChannelHandlerContext ctx, Object evt) {
        if (evt instanceof IdleStateEvent) {
            ctx.channel().close();
        }
    }

    @Override
    public void handlerAdded(ChannelHandlerContext ctx) {
        // Device online
    }

    @Override
    public void handlerRemoved(ChannelHandlerContext ctx) {
        // Device offline
    }

    // Example response to client
    private void sendOk(ChannelHandlerContext ctx) {
        String response = "OK";
        ctx.writeAndFlush(Unpooled.copiedBuffer(response.getBytes(StandardCharsets.UTF_8)));
    }
}

Overall pipeline architecture

TCP Data
  ↓
Custom Decoder (protocol parsing)
  ↓
IdleStateHandler (heartbeat / timeout)
  ↓
Business Handler (device logic)
  ↓
Encoder (optional)

The layered design cleanly separates protocol parsing, heartbeat management, and business processing, resulting in a maintainable solution for custom TCP or IoT communication scenarios.