Mastering Nginx: Installation, Load Balancing, Reverse Proxy, and Performance Tuning

Introduction

Early services were deployed on a single node. As traffic grew, a single server could no longer handle the load, leading to frequent crashes and system paralysis.

Two main problems arise: ① The monolithic deployment cannot sustain increasing traffic. ② When a backend node fails, the whole system becomes unavailable.

Introducing load balancing brings the following benefits:

High availability: traffic can be shifted to other nodes when one fails.

High performance: multiple servers share the load, increasing throughput.

Scalability: nodes can be added or removed as needed.

There are two main load‑balancing approaches: hardware‑level (e.g., A10, F5) and software‑level (e.g., Nginx). Hardware solutions are expensive, while software solutions like Nginx are cost‑effective.

1. Nginx Overview

Nginx is a lightweight, high‑performance HTTP reverse proxy that also supports TCP, UDP, SMTP, HTTPS, and many other protocols. It shares the same event‑driven, multi‑process architecture as Redis, offering low resource consumption and high concurrency (up to 5W connections in theory).

Below is a simple Nginx installation workflow.

# Create directories
mkdir /soft && mkdir /soft/nginx/
cd /soft/nginx/
# Download source
wget https://nginx.org/download/nginx-1.21.6.tar.gz
# Extract
tar -xvzf nginx-1.21.6.tar.gz
# Install dependencies
yum install -y gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel
# Configure and compile
./configure --prefix=/soft/nginx/
make && make install
# Verify installation
sbin/nginx -v

2. Nginx Reverse Proxy & Load Balancing

After installing Nginx, configure an upstream group and a proxy rule:

# /soft/nginx/conf/nginx.conf
upstream nginx_boot {
    server 192.168.0.000:8080 weight=100 max_fails=2 fail_timeout=30s;
    server 192.168.0.000:8090 weight=200 max_fails=2 fail_timeout=30s;
}
server {
    listen 80;
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://nginx_boot;
    }
}

When a client accesses http://<em>IP</em>:80/, Nginx forwards the request to the upstream servers according to the configured weight (e.g., 2:1 for ports 8080 and 8090).

3. Static Resource Separation (Static/Dynamic Separation)

Static files (JS, CSS, images) should be served directly by Nginx to reduce backend load. Example configuration:

location ~* \.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
    root /soft/nginx/static_resources;
    expires 7d;
}

This ensures that static assets are cached by browsers and do not hit the application servers.

4. Resource Compression

Enable gzip compression to reduce bandwidth:

http {
    gzip on;
    gzip_types text/plain application/javascript text/css application/xml image/jpeg image/gif image/png;
    gzip_comp_level 5;
    gzip_vary on;
    gzip_buffers 16 8k;
    gzip_disable "MSIE [1-6]\\.";
    gzip_http_version 1.1;
    gzip_min_length 2k;
    gzip_proxied off;
}

5. Buffer Settings

Buffers smooth out differences between client and backend speeds:

http {
    proxy_connect_timeout 10;
    proxy_read_timeout 120;
    proxy_send_timeout 10;
    proxy_buffering on;
    client_body_buffer_size 512k;
    proxy_buffers 4 64k;
    proxy_buffer_size 16k;
    proxy_busy_buffers_size 128k;
    proxy_temp_path /soft/nginx/temp_buffer;
}

6. Caching

Configure proxy cache to store responses and reduce backend load:

http {
    proxy_cache_path /soft/nginx/cache levels=1:2 keys_zone=hot_cache:128m inactive=3d max_size=2g;
    server {
        location / {
            proxy_cache hot_cache;
            proxy_cache_valid 200 206 304 301 302 1d;
            proxy_cache_valid any 30m;
            proxy_cache_key $host$uri$is_args$args;
            proxy_cache_min_uses 3;
            proxy_cache_lock on;
            proxy_cache_lock_timeout 3s;
            proxy_no_cache $cookie_nocache $arg_nocache $arg_comment;
            add_header Cache-Status $upstream_cache_status;
        }
    }
}

7. IP Black/White List

Control access using allow and deny. Example files:

# BlocksIP.conf
deny 192.177.12.222;
deny 192.177.44.201;
deny 127.0.0.0/8;

# WhiteIP.conf
allow 192.177.12.222;
allow 192.177.44.201;
allow 127.45.0.0/16;
deny all;

Include them in nginx.conf where needed:

http {
    include /soft/nginx/IP/BlocksIP.conf;
    server {
        location /api/ {
            include /soft/nginx/IP/WhiteIP.conf;
        }
    }
}

8. CORS (Cross‑Origin Resource Sharing)

Add the following headers to allow cross‑origin requests:

location / {
    add_header 'Access-Control-Allow-Origin' '*';
    add_header 'Access-Control-Allow-Credentials' 'true';
    add_header 'Access-Control-Allow-Methods' 'GET,POST,OPTIONS,PUT';
    add_header 'Access-Control-Allow-Headers' '*';
    add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range';
    if ($request_method = 'OPTIONS') {
        add_header 'Access-Control-Max-Age' 1728000;
        add_header 'Content-Type' 'text/plain; charset=utf-8';
        add_header 'Content-Length' 0;
        return 204;
    }
}

9. Anti‑Hotlinking

Prevent other sites from embedding your resources:

location ~* \.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
    valid_referers none blocked server_names 192.168.12.129;
    if ($invalid_referer) {
        return 403;
    }
    root /soft/nginx/static_resources;
    expires 7d;
}

10. Large File Transfer

Adjust timeouts and buffer sizes for big files:

client_max_body_size 1g;
client_header_timeout 60s;
client_body_timeout 60s;
proxy_read_timeout 300s;
proxy_send_timeout 300s;

11. SSL Configuration

Obtain a certificate (.crt/.key/.pem) and configure HTTPS:

# HTTPS server
server {
    listen 443 ssl;
    server_name www.example.com;
    ssl_certificate /soft/nginx/certificate/example.pem;
    ssl_certificate_key /soft/nginx/certificate/example.key;
    ssl_session_timeout 5m;
    ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    location / {
        # proxy or static content
    }
}
# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name www.example.com;
    return 301 https://$host$request_uri;
}

12. High Availability with Keepalived

Deploy two Nginx nodes with a virtual IP (VIP) managed by Keepalived. Sample keepalived.conf:

global_defs {
    router_id 192.168.12.129;
}

vrrp_script check_nginx {
    script "/soft/scripts/keepalived/check_nginx.sh";
    interval 3;
    weight -20;
}

vrrp_instance VI_1 {
    state MASTER;
    interface ens33;
    virtual_router_id 121;
    mcast_src_ip 192.168.12.129;
    priority 100;
    nopreempt;
    advert_int 1;
    authentication {
        auth_type PASS;
        auth_pass 1111;
    }
    track_script { check_nginx }
    virtual_ipaddress {
        192.168.12.111;
    }
}

On the backup node, set state BACKUP and a lower priority. Use a script to restart Nginx if its process disappears, and let Keepalived move the VIP to the backup.

13. Performance Optimizations

Enable keepalive connections:

keepalive 32;

keepalive_requests 100;

keepalive_timeout 60s;

Turn on zero‑copy: sendfile on; Configure TCP options: tcp_nodelay on; for low‑latency services, or tcp_nopush on; for high‑throughput static file delivery.

Set worker_processes auto; and increase worker_rlimit_nofile 20000; Bind workers to CPUs: worker_cpu_affinity auto; Use epoll and raise worker_connections 10240; These settings together can significantly improve Nginx throughput and latency.