Master Web Scraping in Python with urllib2: A Step‑by‑Step Guide

Web scraping means reading network resources specified by a URL from the web stream and saving them locally, similar to simulating a browser's HTTP request and reading the server's response.

In Python, the urllib2 module provides a simple interface for fetching web pages via the urlopen function.

The simplest urllib2 program consists of four lines. A file named urllib2_test01.py demonstrates this basic usage:

Running the script (press F5) prints the HTML source of the Baidu homepage, which matches the source viewed via a browser's "View Page Source" feature.

Besides the http: scheme, urllib2 also supports ftp:, file:, and other protocols.

urllib2 uses a Request object to represent an HTTP request. By creating a Request with the target URL and passing it to urlopen, you receive a response object that behaves like a file, allowing you to call .read() to obtain the content.

A second example in urllib2_test02.py shows the same output as the first script:

urllib2 handles all URL schemes with the same interface, allowing you to create an FTP request in a similar way:

When making HTTP requests, you can also send additional data. POST requests are commonly used to submit form data, but you can POST any data to your own programs. Data must be URL‑encoded using functions from the urllib module (not urllib2) before being passed as the data parameter of the Request object.

An example in urllib2_test03.py demonstrates sending data:

If no data parameter is provided, urllib2 performs a GET request. GET and POST differ mainly in side effects: POST typically changes server state, while GET can also transmit data by encoding it in the URL.

Data can also be sent via a GET request by encoding it directly in the URL, as shown:

Some sites dislike programmatic access or serve different content based on the client. By default, urllib2 identifies itself as "Python-urllib/x.y". You can customize the User-Agent header by providing a headers dictionary when constructing the Request object, for example to mimic Internet Explorer.