How to Build an Enterprise‑Grade WAF with OpenResty from Scratch

How to Build an Enterprise‑Grade WAF with OpenResty from Scratch

In today’s internet environment, web applications face increasing security threats. As operations engineers, we often need to implement effective protection on a limited budget. This article shares how to use OpenResty to build a lightweight yet powerful web‑application protection system that has been running stably in production for over two years, blocking tens of millions of malicious requests.

Why Choose OpenResty?

Before diving into implementation, let’s explain why OpenResty is an ideal choice for building a WAF.

OpenResty is a web platform based on Nginx that embeds LuaJIT into Nginx, allowing Lua scripts to extend Nginx functionality. This architecture brings several key advantages:

Excellent Performance : OpenResty inherits Nginx’s high‑performance characteristics, and LuaJIT’s just‑in‑time compilation makes Lua execution speed close to C. In our tests a single machine can handle over 100k QPS while performing complex security checks.

Strong Flexibility : With Lua scripts we can implement various complex security policies, from simple IP black/white lists to sophisticated behavior analysis.

Cost‑Effective : Compared with commercial WAF products that cost hundreds of thousands of dollars in licensing, OpenResty is completely open‑source and free, requiring only development and maintenance effort.

Core Architecture Design

The WAF system uses a modular design with the following core components:

1. Request Lifecycle Management

OpenResty provides multiple execution phases; we need to perform different security checks at different phases:

-- init_by_lua_block: initialization phase, load config and rules
local config = require "waf.config"
local rules = require "waf.rules"

-- initialize shared memory
local limit_req_store = ngx.shared.limit_req_store
local blacklist = ngx.shared.blacklist
local whitelist = ngx.shared.whitelist

-- load security rules
function init_waf()
    -- load SQL injection patterns
    rules.sql_patterns = {
        "select.*from",
        "union.*select",
        "insert.*into",
        "delete.*from",
        "update.*set",
        "drop.*table",
        "exec.*\\(",
        "execute.*\\(",
        "script.*>",
        "javascript:",
        "vbscript:",
        "onload=",
        "onerror=",
        "onclick="
    }
    -- compile regex for performance
    for i, pattern in ipairs(rules.sql_patterns) do
        rules.sql_patterns[i] = ngx.re.compile(pattern, "joi")
    end
end

init_waf()

2. IP Access Control Module

IP access control is the most basic and effective protection. We implement dynamic IP black/white list management:

-- ip_filter.lua
local _M = {}
local blacklist = ngx.shared.blacklist
local whitelist = ngx.shared.whitelist

function _M.check_ip()
    local client_ip = ngx.var.remote_addr
    -- whitelist takes priority
    local is_white = whitelist:get(client_ip)
    if is_white then
        return true
    end
    -- check blacklist
    local is_black = blacklist:get(client_ip)
    if is_black then
        ngx.log(ngx.ERR, "Blocked IP: ", client_ip)
        ngx.status = 403
        ngx.say('{"code": 403, "msg": "Access Denied"}')
        ngx.exit(403)
        return false
    end
    return true
end

function _M.add_blacklist(ip, expire_time)
    expire_time = expire_time or 3600 -- default 1 hour
    local succ, err = blacklist:set(ip, true, expire_time)
    if not succ then
        ngx.log(ngx.ERR, "Failed to add blacklist: ", err)
    end
    return succ
end

return _M

3. Rate Limiting Module

Rate limiting is crucial for defending DDoS and brute‑force attacks. We implement a token‑bucket limiter:

-- rate_limiter.lua
local _M = {}
local limit_req_store = ngx.shared.limit_req_store

function _M.is_limited(key, rate, burst)
    local now = ngx.now()
    local token_key = "tokens:" .. key
    local time_key = "time:" .. key
    local last_time = limit_req_store:get(time_key) or 0
    local tokens = limit_req_store:get(token_key) or burst
    local elapsed = math.max(0, now - last_time)
    local new_tokens = math.min(burst, tokens + elapsed * rate)
    if new_tokens < 1 then
        return true
    end
    limit_req_store:set(token_key, new_tokens - 1)
    limit_req_store:set(time_key, now)
    return false
end

function _M.check_rate_limit()
    local client_ip = ngx.var.remote_addr
    local uri = ngx.var.uri
    local limits = {
        ["/api/login"] = {rate = 1, burst = 5},
        ["/api/register"] = {rate = 0.5, burst = 3},
        ["default"] = {rate = 10, burst = 50}
    }
    local limit = limits[uri] or limits["default"]
    local key = client_ip .. ":" .. uri
    if _M.is_limited(key, limit.rate, limit.burst) then
        ngx.status = 429
        ngx.header["Retry-After"] = "60"
        ngx.say('{"code": 429, "msg": "Too Many Requests"}')
        ngx.exit(429)
    end
end

return _M

4. SQL Injection Detection Module

SQL injection is one of the most common web attacks. We use multi‑layer detection to defend against it:

-- sql_injection.lua
local _M = {}
local sql_patterns = {
    "select.*from",
    "union.*select",
    "insert.*into",
    "delete.*from",
    "update.*set",
    "drop.*table",
    "concat.*\\(",
    "group_concat.*\\(",
    "union.*all",
    "information_schema",
    "sysobjects",
    "syscolumns",
    "--|#|/\\*|\\*/",
    "'.*or.*'='",
    "\".*or.*\"=\"",
    "1=1",
    "1=2",
    "sleep\\s*\\(",
    "benchmark\\s*\\(",
    "waitfor\\s+delay"
}

function _M.check_sql_injection()
    local args = ngx.req.get_uri_args()
    for key, val in pairs(args) do
        if type(val) == "string" and _M.detect_sql_injection(val) then
            _M.block_request("SQL Injection in URL parameter: " .. key)
            return false
        elseif type(val) == "table" then
            for _, v in ipairs(val) do
                if _M.detect_sql_injection(v) then
                    _M.block_request("SQL Injection in URL parameter: " .. key)
                    return false
                end
            end
        end
    end
    ngx.req.read_body()
    local post_args = ngx.req.get_post_args()
    if post_args then
        for key, val in pairs(post_args) do
            if type(val) == "string" and _M.detect_sql_injection(val) then
                _M.block_request("SQL Injection in POST parameter: " .. key)
                return false
            end
        end
    end
    return true
end

function _M.detect_sql_injection(input)
    if not input then return false end
    local lower_input = string.lower(input)
    local decoded = ngx.unescape_uri(lower_input)
    for _, pattern in ipairs(sql_patterns) do
        if ngx.re.find(decoded, pattern, "joi") then
            ngx.log(ngx.WARN, "SQL injection detected: ", pattern, " in: ", input)
            return true
        end
    end
    return false
end

function _M.block_request(reason)
    ngx.log(ngx.ERR, "Blocked request: ", reason)
    ngx.status = 403
    ngx.say('{"code": 403, "msg": "Potential SQL Injection Detected"}')
    ngx.exit(403)
end

return _M

5. XSS Attack Protection Module

Cross‑site scripting (XSS) is another common threat. Our XSS module uses multiple strategies:

-- xss_filter.lua
local _M = {}
local xss_patterns = {
    "on(load|error|click|mouse|key|submit|focus|blur)\\s*=",
    "<script[^>]*>.*</script>",
    "<script[^>]*/>",
    "javascript:\\s*",
    "vbscript:\\s*",
    "<iframe[^>]*>",
    "<object[^>]*>",
    "<embed[^>]*>",
    "<applet[^>]*>",
    "<meta[^>]*>",
    "<link[^>]*>",
    "data:text/html",
    "eval\\s*\\(",
    "expression\\s*\\(",
    "document\\.(cookie|write|location)",
    "window\\.(location|open)",
    "alert\\s*\\(",
    "prompt\\s*\\(",
    "confirm\\s*\\("
}

function _M.check_xss()
    local args = ngx.req.get_uri_args()
    for key, val in pairs(args) do
        if _M.detect_xss(val) then
            _M.block_xss("XSS in URL parameter: " .. key)
            return false
        end
    end
    ngx.req.read_body()
    local post_args = ngx.req.get_post_args()
    if post_args then
        for key, val in pairs(post_args) do
            if _M.detect_xss(val) then
                _M.block_xss("XSS in POST parameter: " .. key)
                return false
            end
        end
    end
    local cookies = ngx.var.http_cookie
    if cookies and _M.detect_xss(cookies) then
        _M.block_xss("XSS in Cookie")
        return false
    end
    local referer = ngx.var.http_referer
    if referer and _M.detect_xss(referer) then
        _M.block_xss("XSS in Referer")
        return false
    end
    return true
end

function _M.detect_xss(input)
    if not input then return false end
    if type(input) == "table" then
        for _, v in ipairs(input) do
            if _M.detect_xss(v) then return true end
        end
        return false
    end
    local decoded = ngx.unescape_uri(input)
    local lower = string.lower(decoded)
    for _, pattern in ipairs(xss_patterns) do
        if ngx.re.find(lower, pattern, "joi") then
            ngx.log(ngx.WARN, "XSS detected: ", pattern)
            return true
        end
    end
    return false
end

function _M.block_xss(reason)
    ngx.log(ngx.ERR, "XSS blocked: ", reason)
    ngx.status = 403
    ngx.say('{"code": 403, "msg": "XSS Attack Detected"}')
    ngx.exit(403)
end

return _M

6. Intelligent CC Attack Defense

CC attacks consume server resources with massive seemingly normal requests. We implement behavior‑analysis based detection:

-- cc_defense.lua
local _M = {}
local cc_store = ngx.shared.cc_store

function _M.check_cc_attack()
    local client_ip = ngx.var.remote_addr
    local now = ngx.now()
    local req_key = client_ip .. ":reqs"
    local reqs, _ = cc_store:incr(req_key, 1, 0, 60) -- 60‑second window
    if reqs > 100 then
        if _M.analyze_behavior(client_ip) then
            _M.block_cc(client_ip)
            return false
        end
    end
    return true
end

function _M.analyze_behavior(ip)
    local pattern_key = ip .. ":pattern"
    local patterns = cc_store:get(pattern_key)
    if not patterns then patterns = {} else patterns = cjson.decode(patterns) end
    local ua = ngx.var.http_user_agent or ""
    local uri = ngx.var.uri
    local method = ngx.var.request_method
    table.insert(patterns, {time = ngx.now(), uri = uri, method = method, ua_hash = ngx.md5(ua)})
    if #patterns > 100 then table.remove(patterns, 1) end
    cc_store:set(pattern_key, cjson.encode(patterns), 300)
    -- simple heuristics: many different UAs or overly regular intervals
    local ua_set = {}
    for _, p in ipairs(patterns) do ua_set[p.ua_hash] = true end
    if table_len(ua_set) > 5 then return true end
    local intervals = {}
    for i = 2, #patterns do
        table.insert(intervals, patterns[i].time - patterns[i-1].time)
    end
    if #intervals > 10 then
        local avg = 0
        for _, v in ipairs(intervals) do avg = avg + v end
        avg = avg / #intervals
        local var = 0
        for _, v in ipairs(intervals) do var = var + (v - avg)^2 end
        var = var / #intervals
        if var < 0.01 then return true end
    end
    return false
end

function _M.block_cc(ip)
    ngx.log(ngx.ERR, "CC attack detected from: ", ip)
    local blacklist = ngx.shared.blacklist
    blacklist:set(ip, true, 1800) -- block 30 minutes
    ngx.status = 503
    ngx.say('{"code": 503, "msg": "Service Temporarily Unavailable"}')
    ngx.exit(503)
end

return _M

Configuration Integration

All modules are integrated into the Nginx configuration:

http {
    lua_shared_dict limit_req_store 10m;
    lua_shared_dict blacklist 10m;
    lua_shared_dict whitelist 10m;
    lua_shared_dict cc_store 50m;
    init_by_lua_block {
        require "resty.core"
        ip_filter = require "waf.ip_filter"
        rate_limiter = require "waf.rate_limiter"
        sql_injection = require "waf.sql_injection"
        xss_filter = require "waf.xss_filter"
        cc_defense = require "waf.cc_defense"
    }
    server {
        listen 80;
        server_name example.com;
        access_by_lua_block {
            ip_filter.check_ip()
            rate_limiter.check_rate_limit()
            cc_defense.check_cc_attack()
            sql_injection.check_sql_injection()
            xss_filter.check_xss()
        }
        location / { proxy_pass http://backend; }
        location /waf/admin {
            content_by_lua_block {
                -- admin API (add/remove blacklist, stats, etc.)
            }
        }
    }
}

Monitoring and Alerting

Effective WAF deployment requires monitoring and alerting:

-- monitoring.lua
local _M = {}
function _M.log_attack(attack_type, details)
    local log_data = {
        timestamp = ngx.now(),
        type = attack_type,
        client_ip = ngx.var.remote_addr,
        uri = ngx.var.uri,
        method = ngx.var.request_method,
        user_agent = ngx.var.http_user_agent,
        details = details
    }
    ngx.log(ngx.ERR, cjson.encode(log_data))
    local http = require "resty.http"
    local httpc = http.new()
    httpc:request_uri("http://monitoring-system/api/alert", {
        method = "POST",
        body = cjson.encode(log_data),
        headers = { ["Content-Type"] = "application/json" }
    })
end

function _M.update_stats(attack_type)
    local stats = ngx.shared.stats
    local key = "attack:" .. attack_type .. ":" .. os.date("%Y%m%d")
    stats:incr(key, 1, 0, 86400)
    local count = stats:get(key)
    if count > 1000 then
        _M.send_alert(attack_type, count)
    end
end

function _M.send_alert(attack_type, count)
    ngx.log(ngx.ALERT, "High attack volume detected: ", attack_type, " count: ", count)
    -- implement email/SMS alerting here
end

return _M

Performance Optimization Tips

1. Enable JIT Compilation

require "resty.core"  -- enable LuaJIT FFI acceleration

2. Proper Use of Shared Memory

lua_shared_dict limit_req_store 10m;  # store rate‑limit data
lua_shared_dict blacklist 10m;       # IP blacklist
lua_shared_dict whitelist 5m;        # IP whitelist
lua_shared_dict cc_store 50m;       # CC detection data
lua_shared_dict stats 10m;          # statistics

3. Avoid Blocking Operations

local function async_alert(data)
    ngx.timer.at(0, function()
        local http = require "resty.http"
        local httpc = http.new()
        httpc:request_uri("http://alert-service/notify", { method = "POST", body = cjson.encode(data) })
    end)
end

4. Pre‑compile Regular Expressions

local compiled_rules = {}
for _, pattern in ipairs(raw_patterns) do
    table.insert(compiled_rules, ngx.re.compile(pattern, "joi"))
end
-- later use compiled_rules directly

Real‑World Case Study

During a major sales event, our platform faced three attack waves: brute‑force login attempts, a distributed CC flood, and sophisticated SQL injection attempts. The WAF blocked over 5 million malicious requests in six hours, automatically added suspicious IPs to the blacklist, and required no manual intervention.

Deployment Recommendations

1. Gradual Rollout : Start in monitoring mode, collect data, fine‑tune rules, then enable blocking.

2. Build Whitelists : Include internal services, monitoring tools, and trusted crawlers to avoid false positives.

3. Regular Rule Updates : Security threats evolve; automate rule updates from threat‑intel sources.

4. Prepare Degradation Plans : Have a quick switch to disable WAF features if performance impact becomes critical.

5. Log Analysis and Optimization : Use ELK or similar stacks to analyze WAF logs, identify false‑positives, and continuously improve detection.

Conclusion and Outlook

Using OpenResty we built an enterprise‑grade WAF at minimal cost, achieving robust protection against common web attacks while maintaining high performance and extensibility. Future work includes incorporating machine‑learning for automatic rule generation and exploring container‑native deployments for more flexible security architectures.