Why High Concurrency Isn't About the Network Card: Unveiling the Real Bottlenecks

All Starts From the NIC

High concurrency is a buzzword, but the network card does not distinguish between millions of connections; it only sees electrical signals, so its performance is measured by bandwidth, not concurrency.

Confused

Many developers conflate the IP layer with the transport layer, overlooking that the OS abstracts network I/O as sockets, and that handling sockets efficiently is the real challenge.

No Connection, No Wait

The IP layer is connection‑less, processing each packet independently, whereas sockets are connection‑oriented and require context, leading to higher memory usage and latency.

Thanks to the OS

Linux abstracts all I/O as files, including network I/O as sockets, and provides mechanisms like select and epoll for efficient multiplexing; frameworks such as Nginx, Netty, and Redis rely on epoll for high‑concurrency scenarios.

Core Contradiction

The fundamental bottleneck remains the CPU‑IO mismatch: CPUs advance rapidly per Moore's law, while I/O devices (disk, NIC) improve slowly, causing low CPU utilization.

Interrupts and Caches

CPU‑IO interaction uses interrupts; the OS caches I/O data to reduce costly device accesses, and employs write‑back caching and scheduling algorithms (e.g., elevator algorithm) to improve throughput.

Efficient NIC Utilization

NICs have their own buffers; misconfigured socket backlogs or small NIC buffers cause packet loss. Properly tuning TCP/IP buffers and using epoll for event‑driven reads are essential.

Select vs. Epoll

Select scans all monitored sockets and copies descriptor bitmaps between user and kernel, which is inefficient; epoll avoids these overheads, offering better scalability.

Reactor Multithreaded Model

The Reactor pattern, used by Netty and Tomcat NIO, separates event dispatchers (listening on select/epoll) from event handlers, often backed by thread pools, handling connection and read/write events efficiently.

Nginx Multi‑Process Model

Nginx employs a master‑worker architecture where workers share a listening socket; each worker processes one request at a time without locks, achieving high concurrency with minimal context switching.

Breaking the Bucket Theory

Improving CPU utilization typically raises I/O utilization as well; once I/O is saturated, further CPU gains are meaningless.

Parallel vs. Concurrent

Parallelism leverages multiple cores for true simultaneous execution, while concurrency interleaves tasks on a single core; both are needed for high‑throughput servers.

Multithreaded Design Patterns

Single Threaded Execution : Serialize access using synchronized to avoid race conditions.

Immutable Pattern : Share read‑only data safely across threads.

Guarded Suspension : Use wait/notify to block threads until conditions are met.

Balking : Abort operation early when preconditions fail.

Producer‑Consumer : Separate I/O producers from CPU consumers.

Read‑Write Lock : Allow concurrent reads, exclusive writes.

Future : Convert synchronous calls to asynchronous results.

Message‑Passing Model

Beyond shared memory, the message‑passing paradigm (Actor model and CSP) avoids locks by communicating via immutable messages; Golang’s CSP and Erlang/Akka’s Actor model exemplify this approach.

Conclusion

Understanding the interplay of CPU, OS, and I/O, choosing the right I/O model (epoll, Reactor), and applying appropriate concurrency patterns are key to mastering high‑concurrency systems.