Master Python Web Scraping: 8 Essential urllib2 Techniques

1. Basic page fetching

GET method example using urllib2 to retrieve a web page.

import urllib2
url = "http://www.baidu.com"
response = urllib2.urlopen(url)
print response.read()

POST method

import urllib
import urllib2
url = "http://abcde.com"
form = {'name':'abc','password':'1234'}
form_data = urllib.urlencode(form)
request = urllib2.Request(url, form_data)
response = urllib2.urlopen(request)
print response.read()

2. Using proxy IP

When IPs are blocked, a proxy can be set via ProxyHandler.

import urllib2
proxy = urllib2.ProxyHandler({'http': '127.0.0.1:8087'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
response = urllib2.urlopen('http://www.baidu.com')
print response.read()

3. Cookies handling

The cookielib module provides a CookieJar to manage HTTP cookies together with urllib2.

import urllib2, cookielib
cookie_support = urllib2.HTTPCookieProcessor(cookielib.CookieJar())
opener = urllib2.build_opener(cookie_support)
urllib2.install_opener(opener)
content = urllib2.urlopen('http://XXXX').read()

Manual cookie addition example:

cookie = "PHPSESSID=91rurfqm2329bopnosfu4fvmu7; kmsign=55d2c12c9b1e3; KMUID=b6Ejc1XSwPq9o756AxnBAg="
request.add_header("Cookie", cookie)

4. Pretending to be a browser

Some servers reject non‑browser requests; modify headers like User-Agent and Content-Type to avoid HTTP 403.

import urllib2
headers = {'User-Agent':'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'}
request = urllib2.Request(url='http://my.oschina.net/jhao104/blog?catalog=3463517', headers=headers)
print urllib2.urlopen(request).read()

5. Page parsing

Regular expressions can be used for simple extraction; for robust parsing, use lxml (C‑based, fast, supports XPath) or BeautifulSoup (pure Python, easy to use). Useful links:

Regex tutorial: http://www.cnblogs.com/huxi/archive/2010/07/04/1771073.html

Online regex tester: http://tool.oschina.net/regex/

lxml guide: http://my.oschina.net/jhao104/blog/639448

BeautifulSoup guide: http://cuiqingcai.com/1319.html

6. Captcha handling

Simple captchas can be recognized programmatically; complex ones (e.g., 12306) often require third‑party captcha‑solving services.

7. Gzip compression

Servers may send gzip‑compressed data; indicate support with the Accept-encoding: gzip header and decompress the response.

import urllib2
request = urllib2.Request('http://xxxx.com')
request.add_header('Accept-encoding', 'gzip')
opener = urllib2.build_opener()
f = opener.open(request)

import StringIO, gzip
compresseddata = f.read()
compressedstream = StringIO.StringIO(compresseddata)
gzipper = gzip.GzipFile(fileobj=compressedstream)
print gzipper.read()

8. Multithreaded concurrent fetching

A simple thread‑pool template using Queue and threading to process tasks concurrently.

from threading import Thread
from Queue import Queue
from time import sleep
q = Queue()
NUM = 2
JOBS = 10
def do_something_using(arguments):
    print arguments
def working():
    while True:
        arguments = q.get()
        do_something_using(arguments)
        sleep(1)
        q.task_done()
for i in range(NUM):
    t = Thread(target=working)
    t.setDaemon(True)
    t.start()
for i in range(JOBS):
    q.put(i)
q.join()