Master Python Web Scraping with urllib2: GET, POST, Proxies, and Multithreading

Introduction

After more than a year of using Python, the most common scenarios are rapid web development, crawling, and automation. This article consolidates reusable patterns for building web crawlers.

1. Basic Page Fetching

GET request

import urllib2
url = "http://www.baidu.com"
response = urllib2.urlopen(url)
print response.read()

POST request

import urllib, urllib2
url = "http://abcde.com"
form = {'name':'abc','password':'1234'}
form_data = urllib.urlencode(form)
request = urllib2.Request(url, form_data)
response = urllib2.urlopen(request)
print response.read()

2. Using Proxy IP

When an IP is blocked, a proxy can be set via ProxyHandler:

import urllib2
proxy = urllib2.ProxyHandler({'http': '127.0.0.1:8087'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
response = urllib2.urlopen('http://www.baidu.com')
print response.read()

3. Cookie Handling

Cookies are managed with cookielib.CookieJar and HTTPCookieProcessor:

import urllib2, cookielib
cookie_support = urllib2.HTTPCookieProcessor(cookielib.CookieJar())
opener = urllib2.build_opener(cookie_support)
urllib2.install_opener(opener)
content = urllib2.urlopen('http://XXXX').read()

Manual cookie addition example:

cookie = "PHPSESSID=91rurfqm2329bopnosfu4fvmu7; kmsign=55d2c12c9b1e3; KMUID=b6Ejc1XSwPq9o756AxnBAg="
request.add_header("Cookie", cookie)

4. Spoofing a Browser

Some sites reject crawlers; modify request headers such as User-Agent and Content-Type to appear as a browser:

import urllib2
headers = {'User-Agent':'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'}
request = urllib2.Request(url='http://my.oschina.net/jhao104/blog?catalog=3463517', headers=headers)
print urllib2.urlopen(request).read()

5. Page Parsing

Regular expressions are powerful but site‑specific. For robust parsing, use lxml (C‑based, fast, supports XPath) or BeautifulSoup (pure Python, easy to use). Helpful resources are linked in the original article.

6. Captcha Handling

Simple captchas can be recognized programmatically; complex ones (e.g., 12306) often require third‑party solving services.

7. Gzip Compression

To receive compressed data, add an Accept-encoding: gzip header and decompress the response:

import urllib2, httplib
request = urllib2.Request('http://xxxx.com')
request.add_header('Accept-encoding', 'gzip')
opener = urllib2.build_opener()
f = opener.open(request)

import StringIO, gzip
compresseddata = f.read()
compressedstream = StringIO.StringIO(compresseddata)
gzipper = gzip.GzipFile(fileobj=compressedstream)
print gzipper.read()

8. Multithreaded Crawling

A simple thread‑pool template demonstrates concurrent fetching:

from threading import Thread
from Queue import Queue
from time import sleep
q = Queue()
NUM = 2
JOBS = 10

def do_something(arg):
    print arg

def worker():
    while True:
        arg = q.get()
        do_something(arg)
        sleep(1)
        q.task_done()

for i in range(NUM):
    t = Thread(target=worker)
    t.setDaemon(True)
    t.start()

for i in range(JOBS):
    q.put(i)

q.join()