Master Python Web Scraping: GET, POST, Proxies, Cookies, and Multithreading

1. Basic Page Retrieval

GET method example:

import urllib2
url = "http://www.baidu.com"
response = urllib2.urlopen(url)
print response.read()

POST Method

import urllib
import urllib2

url = "http://abcde.com"
form = {'name':'abc','password':'1234'}
form_data = urllib.urlencode(form)
request = urllib2.Request(url, form_data)
response = urllib2.urlopen(request)
print response.read()

2. Using Proxy IP

import urllib2

proxy = urllib2.ProxyHandler({'http': '127.0.0.1:8087'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
response = urllib2.urlopen('http://www.baidu.com')
print response.read()

3. Cookie Handling

import urllib2, cookielib

cookie_support = urllib2.HTTPCookieProcessor(cookielib.CookieJar())
opener = urllib2.build_opener(cookie_support)
urllib2.install_opener(opener)
content = urllib2.urlopen('http://XXXX').read()

Manual cookie addition example:

cookie = "PHPSESSID=...; kmsign=...; KMUID=..."
request.add_header("Cookie", cookie)

4. Spoofing as a Browser

Set User-Agent and Content-Type headers to avoid 403 errors.

import urllib2

headers = {'User-Agent':'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'}
request = urllib2.Request(url='http://my.oschina.net/jhao104/blog?catalog=3463517', headers=headers)
print urllib2.urlopen(request).read()

5. Page Parsing

Use regular expressions, lxml, or BeautifulSoup for HTML/XML parsing; tutorials are linked for each method.

6. Captcha Handling

Simple captchas can be recognized programmatically; complex captchas may require paid third‑party services.

7. Gzip Compression

import urllib2, httplib

request = urllib2.Request('http://xxxx.com')
request.add_header('Accept-encoding', 'gzip')
opener = urllib2.build_opener()
f = opener.open(request)

import StringIO
import gzip

compresseddata = f.read()
compressedstream = StringIO.StringIO(compresseddata)
gzipper = gzip.GzipFile(fileobj=compressedstream)
print gzipper.read()

8. Multithreaded Crawling

from threading import Thread
from Queue import Queue
from time import sleep

q = Queue()
NUM = 2
JOBS = 10

def do_something_using(arguments):
    print arguments

def working():
    while True:
        arguments = q.get()
        do_something_using(arguments)
        sleep(1)
        q.task_done()

for i in range(NUM):
    t = Thread(target=working)
    t.setDaemon(True)
    t.start()

for i in range(JOBS):
    q.put(i)

q.join()