Master Python's Requests Library: Quick Guide to GET, POST, Proxies, Sessions & SSL

Requests Library

Although Python's standard library urllib provides most HTTP functionality, its API can be cumbersome; Requests advertises itself as “HTTP for Humans”, offering a simpler, more convenient interface.

Installation and Documentation

Install via pip: pip install requests Documentation: http://docs.python-requests.org/zh_CN/latest/index.html GitHub: https://github.com/requests/requests

Sending GET Requests

Basic GET request:

response = requests.get("http://www.baidu.com/")

Adding headers and query parameters:

import requests
kw = {'wd': '中国'}
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"}
# params accepts a dict or string; dict is automatically URL‑encoded
response = requests.get("http://www.baidu.com/s", params=kw, headers=headers)
print(response.text)
print(response.content)
print(response.url)
print(response.encoding)
print(response.status_code)

Sending POST Requests

Basic POST request:

response = requests.post("http://www.baidu.com/", data=data)

Sending form data without manual URL‑encoding, e.g., posting to Lagou:

import requests
url = "https://www.lagou.com/jobs/positionAjax.json?city=深圳&needAddtionalResult=false&isSchoolJob=0"
headers = {
    'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36",
    'Referer': "https://www.lagou.com/jobs/list_python?labelWords=&fromSearch=true&suggestion='"
}
data = {
    'first': 'true',
    'pn': 1,
    'kd': 'python'
}
resp = requests.post(url, headers=headers, data=data)
print(resp.json())

Using Proxies

Pass a proxies dictionary to the request:

import requests
url = "http://httpbin.org/get"
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36"}
proxy = {"http": "171.14.209.180:27829"}
resp = requests.get(url, headers=headers, proxies=proxy)
with open('xx.html', 'w', encoding='utf-8') as fp:
    fp.write(resp.text)

Cookies

Access cookies returned in a response via the cookies attribute:

import requests
url = "http://www.renren.com/PLogin.do"
data = {"email": "[email protected]", "password": "pythonspider"}
resp = requests.get('http://www.baidu.com/')
print(resp.cookies)
print(resp.cookies.get_dict())

Session Objects

Use a Session to persist cookies across multiple requests:

import requests
url = "http://www.renren.com/PLogin.do"
data = {"email": "[email protected]", "password": "pythonspider"}
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36"}
session = requests.session()
session.post(url, data=data, headers=headers)
resp = session.get('http://www.renren.com/880151247/profile')
print(resp.text)

Handling Untrusted SSL Certificates

Disable verification for sites with untrusted SSL certificates:

resp = requests.get('http://www.12306.cn/mormhweb/', verify=False)
print(resp.content.decode('utf-8'))