How to Scrape Free Proxy IPs with XPath and Python – Step‑by‑Step Guide

Introduction

Many beginners find Python web scraping difficult, especially when constructing regular expressions and dealing with IP bans. This article shows how to use XPath to scrape free proxy IPs from kuaidaili.com, avoiding eye‑strain and IP blocking.

What is XPath?

XPath is an XML Path Language that allows navigation of XML/HTML documents using path expressions. It provides a standard function library, is a core element of XSLT, and is a W3C standard.

Purpose of XPath

Identify the location of nodes in an XML document.

Search for information within XML/HTML.

Navigate elements and attributes.

XPath includes over 100 built‑in functions for strings, numbers, dates, nodes, sequences, logical values, etc., and can be used in Python crawlers for data extraction.

Common XPath Syntax

Typical expressions include: nodename – select all child nodes of this node. / – select from the root node. // – select nodes in the document from the current node regardless of location. . – select the current node. .. – select the parent of the current node. @ – select an attribute.

Example rule: //title[@*] selects all title elements that have any attribute.

Installing and Using XPath Helper

Installation

Search “XPath Helper” in the Chrome Web Store and add the extension. Pin the extension for quick access.

Usage

Open the target website (kuaidaili.com free proxy page), open Developer Tools, locate the desired node, launch XPath Helper, and input the XPath expression to verify the selected data.

Practical Walkthrough

Scraping the Home Page

Observe that the page number appears at the end of the URL; use it as a page parameter for pagination.

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36'
}

def get_page(page):
    url = 'https://www.kuaidaili.com/free/inha/' + str(page)
    response = requests.get(url, headers=headers)
    # Convert response text to a Selector for XPath
    html = parsel.Selector(response.text)
    parse_page(html)

The function builds request headers, fetches the page, converts the HTML to a parsel.Selector, and calls parse_page.

Extracting Content with XPath

Identify the table node that contains the proxy data and test the XPath expression with the helper.

After confirming the range, the XPath rule extracts IP addresses; replace “IP” with “PORT” to get ports.

def parse_page(html):
    # XPath match range
    parse_list = html.xpath('//table[@class="table table-bordered table-striped"]/tbody/tr')
    for tr in parse_list:
        parse_dict = {}
        http = tr.xpath('./td[@data-title="类型"]//text()').extract_first()
        ip = tr.xpath('./td[@data-title="IP"]//text()').extract_first()
        port = tr.xpath('./td[@data-title="PORT"]//text()').extract_first()
        parse_dict[http] = ip + ':' + port
        time.sleep(0.1)
        print(parse_dict)

Key points:

When constructing XPath, replace the leading / with . to limit the search to the current node.

Use text() to retrieve node text. extract_first() returns the first string from the list.

Looping Through Pages

if __name__ == '__main__':
    for page in range(1, 3):
        get_page(page)

This loop crawls the first two pages of free proxies and prints the results.

Result Display

Getting the Code

Readers can request the full script by replying with “代理IP” to the public account.