Getting Started with Python requests: Basic Methods, Parameters, and Session Management

The requests package is a Python library for fetching web content via HTTP, built on urllib3; its official documentation is titled “Requests: HTTP for Humans”.

Basic usage starts with importing the library: import requests Sending a request

The simplest operation is a GET request:

url = 'http://www.zju.edu.cn/'
r = requests.get(url)

The variable r is a Response object that holds all returned data and provides methods to extract needed information. Besides get(), the library also offers post() and many other HTTP methods.

You can retrieve the final URL with r.url.

Reading response content

For textual responses (e.g., HTML), use the text attribute:

r = requests.get(url)
print(r.text)

If the response is binary (e.g., an image), read the raw bytes via content:

r = requests.get(url)
f = open('file', 'wb')
f.write(r.content)

When the response is JSON, requests provides a built‑in parser; call json() to obtain a Python dictionary:

url = 'http://ip.taobao.com/service/getIpInfo.php'
payload = {'ip': '23.91.98.188'}
r = requests.get(url, params=payload)
print(r.json()['data']['country_id'])

This example queries Taobao's IP information service.

Adding query parameters

Instead of manually concatenating a query string, pass a dictionary to the params argument:

url = 'http://www.baidu.com/s'
payload = {'wd': 'Python', 'ie': 'utf-8'}
r = requests.get(url, params=payload)

Adding form data (POST)

For POST requests, supply a dictionary to the data argument:

payload = {'username': 'username', 'password': 'passwd'}
r = requests.post(url, data=payload)

Handling response encoding

Usually requests detects the correct encoding from headers, but you can set it manually via the encoding attribute:

r = requests.get(url)
# optional: r.encoding = 'gb2312'
# write text with the detected or set encoding
f.write(r.text, encoding=r.encoding)

Using a proxy

Provide a dictionary to the proxies argument; the example shows a Shadowsocks SOCKS5 proxy:

proxies = {
    'http': 'socks5://127.0.0.1:1080',
    'https': 'socks5://127.0.0.1:1080'
}
r = requests.get(url, proxies=proxies)

Getting the status code

r = requests.get(url)
print(r.status_code)

Custom request headers

header = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
}
r = requests.get(url, headers=header)

Session objects

Sessions preserve parameters and cookies across multiple requests: s = requests.Session() The session s supports all standard request methods and simplifies handling of cookies and persistent settings.