How to Block Unwanted User Agents in Nginx

First, create a configuration file named agent_deny.conf in the Nginx /usr/local/nginx/conf directory:

# Block Scrapy and similar tools
if ($http_user_agent ~* (Scrapy|Curl|HttpClient)) {
    return 403;
}

# Block specific or empty User‑Agents
if ($http_user_agent ~ "FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms|^$") {
    return 403;
}

# Block non‑GET/HEAD/POST methods
if ($request_method !~ ^(GET|HEAD|POST)$) {
    return 403;
}

After saving the file, insert the following line inside the location / { … } block of your site configuration: include agent_deny.conf; Reload Nginx smoothly with: /usr/local/nginx/sbin/nginx -s reload Test the rules using curl:

# Simulate YisouSpider
curl -I -A 'YisouSpider' http://your-site.com
# Simulate empty User‑Agent
curl -I -A '' http://your-site.com
# Simulate Baiduspider (allowed)
curl -I -A 'Baiduspider' http://your-site.com

The first two commands should return HTTP 403, while the Baiduspider request returns 200, confirming that the blocking configuration works as intended.

The article also lists common unwanted or malicious User‑Agent strings (e.g., FeedDemon, CrawlDaddy, ApacheBench, ZmEu) that can be added to the regular expression for further protection.