Mastering Nginx: How to Build Scalable, High‑Performance Web Services

Introduction

This article is a written version of a 2019 GOPS Shenzhen talk that explains how to systematically understand Nginx within the broader Internet context to solve high‑availability problems of large‑scale distributed networks.

1. Characteristics of Large‑Scale Distributed Clusters

Distributed networks exhibit diverse clients, multi‑layer proxies, multi‑level caches, unpredictable traffic spikes, strict security requirements, and rapid business iteration.

The typical REST architecture includes clients, forward/reverse proxies, and origin servers, with caches serving both upstream and downstream.

2. Nginx and Scalability

Nginx improves scalability through three axes: X (horizontal process scaling without code changes), Y (functional partitioning that may require code refactoring), and Z (user‑attribute‑based routing using variables such as IP or URL). These axes can be combined to address complex real‑world scenarios.

Nginx supports a wide range of protocols—HTTP for downstream clients, UDP/TCP at the transport layer, and application‑layer protocols like gRPC and uWSGI for upstream servers.

3. Nginx and Cluster Performance

Caching in Nginx operates on two dimensions: time and space. Space‑based caching pre‑loads likely requested data, while time‑based caching stores responses for repeated requests, reducing upstream load.

Nginx implements both shared cache (usable by all clients) and private cache (per‑client). Proper handling of HTTP cache headers is essential for correct behavior.

When a cached resource expires, Nginx can serve stale content while revalidating with the upstream server, using directives such as

proxy_cache_use_stale

to maintain performance under heavy traffic.

For large media files, the

slice

module allows Nginx to fetch and cache only the requested byte ranges, avoiding the overhead of downloading entire files.

4. TLS/SSL Handling

Nginx terminates TLS traffic from downstream clients and can re‑encrypt traffic to upstream servers, or convert HTTP to HTTPS using certificates defined via variables.

Performance gains come from efficient code, session caching, and ticket‑based resumption, though both strategies have replay‑attack risks; upgrading to TLS 1.3 is recommended.

5. Clever Use of Nginx Modules

Nginx modules fall into four categories: request‑processing, filter, variable‑only, and load‑balancing modules. They follow a uniform pipe‑and‑filter architecture, executing in a defined sequence of 11 processing phases.

Variables are provided by modules and used by others; they are grouped into HTTP request, TCP connection, internal, response, and system categories.

6. OpenResty and Lua Integration

OpenResty bundles Nginx with the LuaJIT engine, providing a rich ecosystem of Lua modules and an SDK that includes cosocket networking, shared dictionaries, timers, coroutine‑based concurrency, request/response manipulation, sub‑requests, and utility libraries (regex, logging, encoding, etc.).

Lua code runs within Nginx's asynchronous C framework via

ngx_http_lua_module

and

ngx_stream_lua_module

, allowing seamless integration of high‑performance scripting.