How to Block Scrapers with Nginx: Simple IP Deny Rules

Data scraping is a persistent issue; while many want to collect others' content, they also wish to protect their own. This article shows how to use Nginx to block IPs and prevent unwanted scraping, with optional iptables alternatives.

1. Find IPs to block

awk '{print $1}' nginx.access.log | sort | uniq -c | sort -n

The command processes nginx.access.log and outputs each IP with its request count. Identify IPs with high request numbers that are not search engine spiders; for example, block 165.91.122.67.

2. Create a block file

Create blockip.conf in the Nginx installation directory and add the deny rule: deny 165.91.122.67; Save the file.

3. Include the block file in Nginx configuration

Edit nginx.conf and add the include statement in the appropriate context (http, server, location, or limit_except): include blockip.conf; 4. Reload Nginx

Run the reload command to apply changes: /usr/local/nginx/nginx -s reload Advanced usage

You can block a single IP, allow a single IP, block all, allow all, or block IP ranges:

deny 192.0.2.1;          # block a single IP
allow 192.0.2.1;         # allow a single IP
deny all;                # block all IPs
allow all;               # allow all IPs
deny 123.0.0.0/8;        # block the 123.0.0.0/8 range
deny 124.45.0.0/16;      # block the 124.45.0.0/16 range
deny 123.45.6.0/24;      # block the 123.45.6.0/24 range

To allow only a few IPs and deny the rest, write in blockip.conf:

allow 1.1.1.1;
allow 1.1.1.2;
den y all;

Place the include blockip.conf; directive inside a specific server{} block to block IPs for a single site, or inside the global http{} block to apply the rule to all sites.