Understanding Netty’s Asynchronous Model, Epoll, and IO Multiplexing: Theory, JNI Integration, and a Hand‑Written Epoll Server

Hello everyone, I am Chen. Many of you may know the "Sword in the Stone" legend; here we treat the reader as the king who can pull the sword, which represents Netty, the powerful epoll‑based networking framework.

Netty's Asynchronous Model

Netty uses a bossGroup and workerGroup driven by ServerBootstrap, combined with configurable handlers to build a clean asynchronous pipeline.

Classic Multithread Model

Each client connection spawns a dedicated thread, which works for a small number of clients but quickly exhausts memory and CPU under heavy load.

Classic Reactor Model

The Reactor model dispatches I/O events to handlers, similar to a telephone exchange.

Reactor: Dispatches events to the appropriate handler.

Handlers: Process the actual I/O business logic.

The diagram shows the Initiation Dispatcher (Reactor) performing event demultiplexing, an Acceptor obtaining resource handles, and EventHandlers handling authentication, heartbeats, business logic, and codec operations.

Reactor Model Evolution

Single‑thread Reactor uses a single selector; pooled Reactor decouples read/write and uses a thread pool; multi‑Reactor splits responsibilities between a mainReactor (accept) and subReactor (read/write), improving CPU utilization.

Netty’s boss/worker groups correspond to the main/sub reactors, and ServerBootstrap creates the Reactor, initializing group, channel, and options.

Netty’s Strength via JNI

Netty achieves native‑level performance by using JNI to call C code for low‑level I/O. Below is a simple Java class that loads a native library and declares a native method.

public class DataSynchronizer {
    static { System.loadLibrary("synchronizer"); }
    private native String syncData(String status);
    public static void main(String... args) {
        String rst = new DataSynchronizer().syncData("ProcessStep2");
        System.out.println("The execute result from C is : " + rst);
    }
}

Compilation steps on Linux Mint:

apt install default-jdk
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

Generate header:

javac -h . DataSynchronizer.java

Implement the native method in C:

#include
#include
#include "DataSynchronizer.h"
JNIEXPORT jstring JNICALL Java_DataSynchronizer_syncData(JNIEnv *env, jobject obj, jstring str) {
    const char *inCStr = (*env)->GetStringUTFChars(env, str, NULL);
    if (NULL == inCStr) return NULL;
    printf("In C, the received string is: %s\n", inCStr);
    (*env)->ReleaseStringUTFChars(env, str, inCStr);
    char outCStr[128];
    printf("Enter a String: ");
    scanf("%s", outCStr);
    return (*env)->NewStringUTF(env, outCStr);
}

Compile the native library:

gcc -fPIC -I"$JAVA_HOME/include" -I"$JAVA_HOME/include/linux" -shared -o libsynchronizer.so DataSynchronizer.c

Run the program:

java -Djava.library.path=. DataSynchronizer

The output shows successful JNI invocation, which is also how Netty accesses the native epoll implementation in the transport-native-epoll module.

IO Multiplexing Models

All file descriptors (including sockets, files, devices) are represented as integers in Linux. The first three descriptors (0,1,2) are stdin, stdout, stderr.

Select Model

Classic API:

int select(int nfds, fd_set *readfds, fd_set *writefds, fd_set *errorfds, struct timeval *timeout);

Example server using select is provided with detailed comments.

Poll Model

API:

int poll(struct pollfd *fds, nfds_t nfds, int timeout);

Uses an array of struct pollfd instead of a fixed‑size bitmap, removing the 1024‑fd limit.

epoll Model

Since Linux 2.6, epoll provides edge‑triggered and level‑triggered notifications with O(1) scalability.

Key functions:

epoll_create – creates an epoll instance.

epoll_ctl – add, modify, or delete file descriptors.

epoll_wait – wait for ready events.

epoll stores interest list in a red‑black tree and ready list in a doubly linked list, avoiding full scans.

Level‑Trigger vs Edge‑Trigger

Level‑Trigger (LT) notifies as long as the condition holds; Edge‑Trigger (ET) notifies only on state changes, requiring the application to drain the buffer completely.

Examples show how a buffer of size 10 behaves under LT and ET when receiving short and long strings.

Hand‑Written epoll Server

The following C program implements a minimal epoll server supporting both LT and ET modes, with extensive comments.

#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#define MAX_EVENT_NUMBER 1024
#define BUFFER_SIZE 10
#define ENABLE_ET 0
int SetNonblocking(int fd) { int old = fcntl(fd, F_GETFL); fcntl(fd, F_SETFL, old | O_NONBLOCK); return old; }
void AddFd(int epoll_fd, int fd, bool enable_et) { struct epoll_event event; event.data.fd = fd; event.events = EPOLLIN; if (enable_et) event.events |= EPOLLET; epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &event); SetNonblocking(fd); }
/* LT processing */
void lt_process(struct epoll_event* events, int number, int epoll_fd, int listen_fd) { char buf[BUFFER_SIZE]; for (int i=0;i
LT Mode: it was triggered once!\n"); memset(buf,0,BUFFER_SIZE); int ret = recv(sockfd,buf,BUFFER_SIZE-1,0); if (ret<=0) { close(sockfd); continue; } printf("get %d bytes of content: %s\n",ret,buf); }
 }
}
/* ET processing */
void et_process(struct epoll_event* events, int number, int epoll_fd, int listen_fd) { char buf[BUFFER_SIZE]; for (int i=0;i
ET Mode: it was triggered once\n"); while (1) { memset(buf,0,BUFFER_SIZE); int ret = recv(sockfd,buf,BUFFER_SIZE-1,0); if (ret < 0) { if (errno==EAGAIN || errno==EWOULDBLOCK) { printf("-->wait to read!\n"); break; } close(sockfd); break; } else if (ret==0) { close(sockfd); break; } else { printf("get %d bytes of content: %s\n",ret,buf); }
 }
 }
 }
}
int main(int argc, char* argv[]) {
    const char* ip = "10.0.76.135"; int port = 9999;
    struct sockaddr_in address; bzero(&address,sizeof(address)); address.sin_family = AF_INET; inet_pton(AF_INET,ip,&address.sin_addr); address.sin_port = htons(port);
    int listen_fd = socket(PF_INET,SOCK_STREAM,0); if (listen_fd<0) { perror("socket"); return -1; }
    if (bind(listen_fd,(struct sockaddr*)&address,sizeof(address))==-1) { perror("bind"); return -1; }
    if (listen(listen_fd,5)==-1) { perror("listen"); return -1; }
    struct epoll_event events[MAX_EVENT_NUMBER];
    int epoll_fd = epoll_create(5);
    if (epoll_fd==-1) { perror("epoll_create"); return -1; }
    AddFd(epoll_fd,listen_fd,true);
    while (1) {
        int ret = epoll_wait(epoll_fd,events,MAX_EVENT_NUMBER,-1);
        if (ret<0) { perror("epoll_wait"); break; }
        if (ENABLE_ET) et_process(events,ret,epoll_fd,listen_fd);
        else lt_process(events,ret,epoll_fd,listen_fd);
    }
    close(listen_fd);
    return 0;
}

The server demonstrates how to set non‑blocking mode, add file descriptors to epoll, and handle both LT and ET events correctly.

Performance Notes

Using ET mode with non‑blocking sockets yields the best scalability for millions of concurrent connections, while LT is simpler but incurs more kernel‑user transitions.

Further tuning (kernel parameters, VM settings, NIC offloading) is required to reach true million‑connection performance, which will be covered in future articles.

References

https://www3.ntu.edu.sg/home/ehchua/programming/java/JavaNativeInterface.html

https://copyconstruct.medium.com/the-method-to-epolls-madness-d9d2d6378642