How to Scrape and Extract Proxy Data with Python: Step-by-Step Guide

1. Introduction

After a previous article on using Python to crawl proxy data, this guide focuses on analyzing the web page structure and extracting the required information.

2. Home Page Analysis and Extraction

The homepage shows a pagination pattern where the number after the URL indicates the page. Each page contains over 100 entries, and the site has more than 2,700 pages, totaling over 270,000 proxy records. To keep the dataset recent, only the first 100 pages are targeted.

URL pattern for the first 100 pages:

http://example.com/page/1
http://example.com/page/2
... 
http://example.com/page/100

3. Web Element Analysis and Extraction

The proxy information is stored inside a <table id="ip_list"> element. The required fields are IP address, port, server location, and type. A ProxyBean class is defined to hold these attributes.

Extraction is performed using regular expressions combined with BeautifulSoup . First, the entire table is captured:

<table id="ip_list">([\S\s]*)</table>

Then each row ( <tr>) is processed. Rows with class="odd" are distinguished from others.

Field‑specific regular expressions:

IP address: (2[0-5]{2}|[0-1]?\d{1,2})(\.(2[0-5]{2}|[0-1]?\d{1,2})){3} Port: <td>([0-9]+)</td> Location: <a href="([^>]+)">([^<]+)</a> Type: <td>([A-Za-z]+)</td> BeautifulSoup parses the table rows and extracts the text for each column, populating a ProxyBean instance for every proxy entry.

After extraction, the data is written to a file for later use.

4. Summary

The project demonstrates how to:

Use the requests library to fetch web pages.

Apply anti‑scraping techniques such as proxy pools.

Write regular expressions for precise element extraction.

Leverage BeautifulSoup to parse HTML tables and retrieve structured data.

Overall, the tutorial provides a practical example of building a Python web crawler for proxy data.