Designing a Reactor‑Based Asynchronous Network Library in Go

When a service needs to handle massive connections and extremely high concurrency, the Go runtime can become a bottleneck: memory consumption grows, goroutine scheduling overhead increases, and GC pauses may even crash the process. In such cases, rebuilding the server with a classic Reactor network model can alleviate the pressure.

1. Common server‑side network programming models

There are three typical models for managing connections and request processing:

Traditional blocking I/O (one thread/process per connection).

Reactor model – a central event dispatcher uses I/O multiplexing (epoll/kqueue) to handle many connections with a few threads.

Proactor model – the kernel performs asynchronous I/O and notifies the application when data is ready.

The Reactor model can be described as I/O multiplexing + non‑blocking I/O . It separates the responsibilities of accepting connections (MainReactor) and processing read/write events (SubReactor).

2. Go’s native network model

Go’s net package implements a single‑Reactor, multi‑goroutine model: each connection is bound to a netFD and a pollDesc . When a read/write returns EAGAIN , the goroutine is parked in the poller until the fd becomes ready.

3. Requirement background

The author’s team needed a Go‑based gateway that could sustain millions of concurrent connections (the existing C++ gateway runs on dozens of machines). Benchmarks showed that as connections grew, Go’s goroutine‑per‑connection approach caused linear memory growth and long GC pauses, eventually crashing the service.

Existing open‑source Reactor libraries (e.g., netpoll, evio, gnet) were evaluated, but they lacked out‑of‑the‑box HTTP/TLS support and had APIs that did not fit the gateway’s needs.

4. Overall layered design

The proposed library is divided into three layers:

Application layer : EchoServer, HTTPServer, TLSServer, GRPCServer, etc. Implements EventHandler interfaces.

Connection layer : Implements the Reactor core. Consists of a MainReactor (acceptor) and multiple SubReactors (read/write loops). Key data structures are: EventLoop – the reactor instance (main or sub). Poller – wraps epoll/kqueue system calls. Conn – a connection bound to a file descriptor.

Base layer : Provides low‑level syscalls (listen, accept, read, write, epoll_create, epoll_ctl, epoll_wait) and memory management.

Important considerations:

FD race : When a connection is closed, its fd may be reused by a new connection before the SubReactor finishes processing the old fd. The solution is to protect each fd operation with a lock and check the connection’s closed flag before reading/writing.

Load balancing : Distribute new connections from the MainReactor to SubReactors using strategies such as round‑robin, fd‑hash, or least‑connections.

5. Memory management strategies

Three typical buffer schemes are compared:

Fixed array : Allocate a fixed‑size buffer for each read – simple but creates many temporary objects.

RingBuffer : Separate read/write buffers, reduces memory usage but may suffer from costly expansions.

LinkBuffer : Nodes store pointers to user buffers, enabling zero‑copy reads/writes and easy scaling. This is the approach used by ByteDance’s netpoll.

Zero‑copy read example:

func (c *Conn) Read() ([]byte, error) { /* returns a slice that directly references the kernel‑filled buffer */ }

Zero‑copy write example:

func (c *Conn) Write(data []byte) error { /* wraps the user slice into a node without copying */ }

6. Performance testing

Two benchmark scenarios were built:

Echo : Simple echo server without protocol parsing.

HTTP : HTTP/1.x server with a 100 k loop to simulate business logic.

Results (summarized):

In low‑connection scenarios, Go’s native net library performs comparably.

When connections exceed 300k–500k, Go’s memory usage and GC time increase dramatically, causing service downtime.

The custom Reactor library (named wnet) remained stable up to 1 million connections, with modest GC overhead.

These findings indicate that for most applications Go’s native networking is sufficient, but in extreme high‑concurrency cases (e.g., flash sales, live‑streaming chat, massive WebSocket connections) a Reactor‑based library can provide the necessary scalability.

7. Conclusion

Go’s native network stack trades memory for latency and throughput. When memory becomes a constraint under massive connections, the Reactor model shines by reusing a small set of threads and avoiding per‑connection goroutine overhead. The article’s design and implementation details can guide developers to build or adapt existing open‑source Reactor libraries (gnet, gev, evio) for their own high‑performance services.