php Courses

Mar 31, 2023 · Backend Development

Blocking AhrefsBot and Scrapy Crawlers with Nginx in Docker

This article describes how to identify abnormal Nginx requests caused by bots like AhrefsBot and Scrapy, and provides a step‑by‑step Docker‑based solution that includes a custom Nginx configuration, agent‑deny rules, and optional Alibaba Cloud security‑group IP blocking to prevent memory spikes.