How to Build a High-Performance Proxy IP Pool for Web Crawlers with Python

In a distributed deep‑web crawling environment, a stable proxy pool service can provide thousands of spiders with valid proxies, ensuring fast and reliable operation. This guide shows how to build a simple proxy pool using free resources.

1. Problems

Where do proxy IPs come from? Beginners often scrape free proxy sites such as Xici or KuaiDaili. Free proxy collection is straightforward: fetch pages → extract with regex/XPath → store.

How to guarantee proxy quality? Most free proxies are unreliable. A detection program should repeatedly test each proxy against a stable website, using multithreading or async methods because testing is slow.

How to store collected proxies? A high‑performance NoSQL database like SSDB (a Redis alternative) is recommended for queues, hashes, sets, and key‑value pairs, supporting terabyte‑scale data.

How to make proxies easy for spiders to use? Expose the pool as a service (e.g., a Flask API). Spiders can request, delete, or refresh proxies via HTTP endpoints, allowing dynamic management.

2. Proxy Pool Design

The pool consists of four parts:

ProxyGetter : fetches the latest free proxies from five sources and inserts them into the DB; additional sources can be added.

DB : stores proxy IPs; SSDB is used as the backend.

Schedule : periodically checks proxy availability, removes invalid ones, and triggers ProxyGetter to refresh the pool.

ProxyApi : external Flask API offering get/delete/refresh operations for spiders.

3. Code Modules

The Python implementation is split into six modules:

Api : Flask‑based API handling get/delete/refresh/get_all calls.

DB : SSDB access layer using a factory pattern for future database extensions.

Manager : core logic for get/delete/refresh/get_all; can be extended to bind proxies with spiders or accounts.

ProxyGetter : crawls five free‑proxy sites (KuaiDaili, 66IP, YouDaili, Xici, guobanjia) and stores results.

Schedule : multi‑process task that periodically refreshes and validates proxies.

Util : common utilities such as configuration parsing, singleton implementation, and lazy property handling.

Additional files include Config.ini for database and proxy‑source configuration, and other supporting scripts.

4. Installation

git clone [email protected]:jhao104/proxy_pool.git
# or download the zip from https://github.com/jhao104/proxy_pool

Install dependencies: pip install -r requirements.txt Start the components (configure SSDB in Config.ini first):

# In the Schedule directory
python ProxyRefreshSchedule.py
# In the Api directory
python ProxyApi.py

5. Usage

After the scheduler runs, it fetches proxies, validates them, and stores the usable ones in SSDB (default refresh every 20 minutes). The API can be accessed via a browser:

Example Python client for the API:

import requests

def get_proxy():
    return requests.get("http://127.0.0.1:5000/get/").content

def delete_proxy(proxy):
    requests.get("http://127.0.0.1:5000/delete/?proxy={}".format(proxy))

def spider():
    # ... your spider logic ...
    requests.get('https://www.example.com', proxies={"http": "http://{}".format(get_proxy)})
    # ...

6. Conclusion

The current implementation is simple and may need further improvements, but it provides a functional proxy pool that can be starred on GitHub. Project URL: https://github.com/jhao104/proxy_pool .