Master High‑Performance Linux Socket Programming with OpenResty and epoll

Socket Programming

In Linux socket programming for handling massive connection requests, non‑blocking I/O and I/O multiplexing mechanisms such as select, poll, and epoll are used. Since the introduction of epoll in Linux 2.6, high‑performance servers like Nginx rely on epoll for high concurrency.

For high‑performance services the focus is not language speed but caching and asynchronous non‑blocking support.

Cache

Cache hierarchy: memory > SSD > mechanical disk; local machine > network; intra‑process > inter‑process. The goal is to maximize hit rate inside the process, which yields the highest overall efficiency.

Asynchronous non‑blocking

When accessing databases, networks, or slow I/O devices, use an event‑driven model so the system notifies the application when the operation completes, allowing the CPU to serve other client connections.

OpenResty

OpenResty is a high‑performance web platform based on Nginx and Lua, integrating refined Lua libraries, third‑party modules, and dependencies. It enables building dynamic web applications that can handle 10K‑1M concurrent connections by running Lua scripts inside Nginx’s non‑blocking I/O model.

OpenResty essentially combines Nginx and LuaJIT, allowing Lua scripts to call C and Lua modules for rapid development.

Because Nginx uses a master‑worker model, a master process manages multiple worker processes. Each worker runs its own Lua VM; when a request arrives, a coroutine is created inside the Lua VM to handle the request. Coroutines share global variables ( _G) but have isolated stacks and local variables, similar to threads but with cooperative scheduling.

OpenResty performance can approach or even exceed native Nginx C modules.

OpenResty Architecture

Load balancing: LVS + HAProxy forwards traffic to core Nginx instances.

Single‑machine closed loop: all required data is available locally, avoiding network hops.

Distributed closed loop: addresses data inconsistency and storage bottlenecks by using master‑slave replication or sharding across multiple servers.

The access gateway (entry layer) receives traffic and performs tasks such as SSL termination, request routing, and load balancing.

OpenResty Environment Setup

Download: http://openresty.org

Download source: http://openresty.org/cn/download.html

Before installation, ensure the following packages are present: perl, libpcre, libssl.

# Check required libraries
$ sudo ldconfig -v

# Install required libraries
$ sudo apt install libpcre3-dev libssl-dev perl make build-essential curl libreadline-dev libncurses5-dev

Download, extract, and configure OpenResty:

$ wget https://openresty.org/download/ngx_openresty-1.13.6.1.tar.gz
$ tar -zxvf ngx_openresty-1.13.6.1.tar.gz
$ mv openresty-1.13.6.1 openresty
$ cd openresty
$ ./configure

Compile and install (default location /usr/local/openresty):

$ sudo make && make install
$ cd /usr/local/openresty

Start Nginx and verify:

$ sudo /usr/local/openresty/nginx/sbin/nginx
$ ps -ef | grep nginx
$ service nginx status

If port 80 is already in use, free it:

$ sudo netstat -ntlp | grep 80
$ sudo killall -9 nginx

Edit the Nginx configuration to listen on both IPv4 and IPv6:

$ sudo vim /etc/nginx/conf/nginx.conf
listen 80;
listen [::]:80 ipv6only=on default_server;

Test with curl or a browser:

$ curl 127.0.0.1

Add OpenResty binaries to the user’s PATH:

$ sudo vim ~/.bashrc
export PATH=$PATH:/usr/local/openresty/nginx/sbin
source ~/.bashrc
$ nginx -s reload

Lua Content Handlers

Using content_by_lua:

$ vim /usr/local/openresty/nginx/conf/nginx.conf
location /test {
  default_type text/html;
  content_by_lua 'ngx.say("hello openresty")';
}
# Reload Nginx
$ /usr/local/openresty/nginx/sbin/nginx -s reload

Using content_by_lua_file:

$ vim nginx.conf
location /test {
  content_by_lua_file 'html/test.lua';
}
$ vim ../html/test.lua
ngx.say("hello lua")
$ sudo /usr/local/openresty/nginx/sbin/nginx -s reload
$ curl 127.0.0.1/test
hello lua

Disable Lua code cache during development (note: this hurts performance in production):

$ vim nginx.conf
server {
  lua_code_cache off;
  location /test {
    content_by_lua_file 'html/test.lua';
  }
}
$ sudo /usr/local/openresty/nginx/sbin/nginx -s reload

Simple OpenResty Example

Create a separate work directory and a minimal configuration:

$ mkdir -p ~/openresty/test/logs ~/openresty/test/conf
$ vim ~/openresty/test/conf/nginx.conf
worker_processes 1;
error_log logs/error.log;
events { worker_connections 10224; }
http {
  server {
    listen 8001;
    location / {
      default_type text/html;
      content_by_lua_block {
        ngx.say("hello world");
      }
    }
  }
}
$ nginx -p ~/openresty/test
$ curl 127.0.0.1:8001
hello world

Example of a Lua script that generates an MD5 hash based on a query parameter:

$ vim nginx.conf
location /test {
  content_by_lua_file "lua/test.lua";
}
$ cd .. && mkdir lua && cd lua
$ vim test.lua
local args = ngx.req.get_uri_args()
local salt = args.salt
if not salt then
  ngx.exit(ngx.HTTP_BAD_REQUEST)
end
local md5str = ngx.md5(ngx.time()..salt)
ngx.say(md5str)
$ sudo /usr/local/openresty/nginx/sbin/nginx -s reload
$ curl -i 127.0.0.1/test?salt=lua
HTTP/1.1 200 OK
...
b55b77f75e46b96b11778ca7edfe8d55

When errors occur, check Nginx’s error log:

$ vim nginx/logs/error.log

ngx_lua API

For further reference see the official NGINX Lua module documentation.