Master urllib2: Proxies, Timeouts, Headers, and Advanced HTTP Techniques

Previously we introduced a simple urllib2 tutorial; now we detail a range of practical usage techniques.

1. Setting a Proxy

urllib2 uses the http_proxy environment variable by default. To control the proxy explicitly within a program, create a custom opener instead of relying on the environment.

Using urllib2.install_opener() sets a global opener, which is convenient but limits fine‑grained control such as using multiple proxies simultaneously. A better approach is to call the opener’s open method directly instead of the global urlopen.

2. Configuring Timeout

In Python versions prior to 2.6, urllib2 did not expose a timeout parameter; you had to modify the global socket timeout. Starting with Python 2.6, you can pass a timeout argument to urllib2.urlopen().

3. Adding Specific Headers to a Request

Use a Request object to attach custom headers. Pay special attention to headers that servers validate, such as User-Agent and Content-Type. Common Content-Type values include application/xml, application/json, and application/x-www-form-urlencoded.

4. Handling Redirects

urllib2 automatically follows HTTP 3xx redirects. To detect a redirect, compare the response URL with the request URL. If you need to disable automatic redirects, either use the lower‑level httplib library or subclass HTTPRedirectHandler.

5. Accessing Cookies

urllib2 handles cookies automatically. To retrieve a specific cookie value, use the following pattern:

Running the script prints the cookie received from Baidu.

6. Using HTTP PUT and DELETE

urllib2 only supports GET and POST. To issue PUT or DELETE requests, you must fall back to the lower‑level httplib library, or craft a custom request as shown:

7. Retrieving HTTP Status Codes

For a 200 OK response, call response.getcode(). For other status codes, urlopen raises an exception; inspect the exception’s code attribute to obtain the status.

8. Enabling Debug Logging

Activate urllib2 debug logging to print request and response details, which helps troubleshoot without external packet capture tools.

The output shows the raw HTTP traffic.

9. Handling Forms

To submit a login form, capture the required fields (e.g., username, password, continueURI, fk, login_submit) using a tool like Firefox + HttpFox, then construct a POST request with those parameters.

10. Spoofing a Browser

Some sites block crawlers. To appear as a regular browser, modify the request headers (e.g., User-Agent) accordingly.

11. Bypassing Anti‑Hotlink Measures

Sites may check the Referer header to prevent hotlinking. Set the Referer to the target site’s URL (e.g., cnbeta) to bypass the check.

The headers dictionary can include any desired header, such as modifying X-Forwarded-For to mask the real IP address.