Comprehensive Guide to Using Python's requests Library for Various HTTP Request Scenarios

1. Send JSON data

If you need to send JSON data, use the json parameter, which automatically sets the Content-Type header to application/json.

import requests
import json
url = 'http://example.com/api/endpoint'
data = {
    "key": "value",
    "another_key": "another_value"
}
headers = {'Content-Type': 'application/json'}
response = requests.post(url, data=json.dumps(data), headers=headers)
print(response.status_code)
print(response.json())

2. Send form data (application/x-www-form-urlencoded)

Use the data parameter to send form‑encoded data; the library sets the appropriate content type automatically.

import requests
url = 'http://example.com/api/endpoint'
data = {
    "key": "value",
    "another_key": "another_value"
}
response = requests.post(url, data=data)
print(response.status_code)
print(response.text)

3. Upload files (multipart/form-data)

When uploading files, use the files parameter, which sets Content-Type to multipart/form-data.

import requests
url = 'http://example.com/api/upload'
file = {'file': ('image.png', open('image.png', 'rb'))}
data = {
    'biz': 'temp',
    'needCompress': 'true'
}
response = requests.post(url, data=data, files=file)
print(response.status_code)
print(response.text)

4. Send requests with authentication

Provide credentials via the auth tuple for HTTP basic authentication.

import requests
url = 'http://example.com/api/endpoint'
username = 'your_username'
password = 'your_password'
response = requests.get(url, auth=(username, password))
print(response.status_code)
print(response.text)

5. Handle redirects

Control whether redirects are followed with the allow_redirects argument (default is True).

import requests
url = 'http://example.com/api/endpoint'
allow_redirects = False
response = requests.get(url, allow_redirects=allow_redirects)
print(response.status_code)
print(response.history)

6. Set request timeout

Specify a timeout (in seconds) to avoid hanging indefinitely.

import requests
url = 'http://example.com/api/endpoint'
timeout = 5
try:
    response = requests.get(url, timeout=timeout)
    print(response.status_code)
except requests.exceptions.Timeout:
    print("The request timed out")

7. Use cookies

Send or receive cookies via the cookies parameter.

import requests
url = 'http://example.com/api/endpoint'
cookies = {'session': '1234567890'}
response = requests.get(url, cookies=cookies)
print(response.status_code)
print(response.cookies)

8. Custom headers

Add any additional headers you need, such as a custom User‑Agent.

import requests
url = 'http://example.com/api/endpoint'
headers = {
    'User-Agent': 'MyApp/0.0.1',
    'X-Custom-Header': 'My custom header value'
}
response = requests.get(url, headers=headers)
print(response.status_code)
print(response.headers)

9. SSL verification

By default, HTTPS requests verify SSL certificates; you can disable verification with verify=False (not recommended for production).

import requests
url = 'https://example.com/api/endpoint'
# Verify SSL (default)
response = requests.get(url, verify=True)
print(response.status_code)
# Skip verification (insecure)
response = requests.get(url, verify=False)
print(response.status_code)

10. Upload multiple files

Pass a list of file tuples to the files argument to upload several files in one request.

import requests
url = 'http://example.com/api/upload'
files = [
    ('file1', ('image1.png', open('image1.png', 'rb'))),
    ('file2', ('image2.png', open('image2.png', 'rb')))
]
data = {
    'biz': 'temp',
    'needCompress': 'true'
}
response = requests.post(url, data=data, files=files)
print(response.status_code)
print(response.text)

11. Use a proxy

Route requests through a proxy server using the proxies dictionary.

import requests
url = 'http://example.com/api/endpoint'
proxies = {
    'http': 'http://10.10.1.10:3128',
    'https': 'http://10.10.1.10:1080'
}
response = requests.get(url, proxies=proxies)
print(response.status_code)

12. Stream large file downloads

Enable streaming to download large files without loading them entirely into memory.

import requests
url = 'http://example.com/largefile.zip'
response = requests.get(url, stream=True)
with open('largefile.zip', 'wb') as f:
    for chunk in response.iter_content(chunk_size=8192):
        if chunk:
            f.write(chunk)

13. Chunked upload of large files

Read a large file in chunks and upload each chunk separately to avoid memory issues.

import requests
url = 'http://example.com/api/upload'
file_path = 'path/to/largefile.zip'
chunk_size = 8192
with open(file_path, 'rb') as f:
    for chunk in iter(lambda: f.read(chunk_size), b''):
        files = {'file': ('largefile.zip', chunk)}
        response = requests.post(url, files=files)
        if response.status_code != 200:
            print("Error uploading file:", response.status_code)
            break

14. Use a Session object

Maintain persistent parameters (e.g., cookies) across multiple requests with requests.Session().

import requests
s = requests.Session()
# Set a cookie manually
s.cookies['example_cookie'] = 'example_value'
# GET request
response = s.get('http://example.com')
# POST request
data = {'key': 'value'}
response = s.post('http://example.com/post', data=data)
print(response.status_code)

15. Error handling

Wrap requests in try/except blocks to catch HTTP errors, connection problems, timeouts, and other request exceptions.

import requests
url = 'http://example.com/api/endpoint'
try:
    response = requests.get(url)
    response.raise_for_status()
except requests.exceptions.HTTPError as errh:
    print(f"Http Error: {errh}")
except requests.exceptions.ConnectionError as errc:
    print(f"Error Connecting: {errc}")
except requests.exceptions.Timeout as errt:
    print(f"Timeout Error: {errt}")
except requests.exceptions.RequestException as err:
    print(f"OOps: Something Else: {err}")

16. Token‑based authentication

Send an API token in the Authorization header.

import requests
url = 'http://example.com/api/endpoint'
token = 'your_auth_token_here'
headers = {'Authorization': f'Token {token}'}
response = requests.get(url, headers=headers)
print(response.status_code)
print(response.json())

17. OAuth2 authentication

Obtain an OAuth2 access token and include it as a Bearer token in subsequent requests.

import requests
# Get token
token_url = 'http://example.com/oauth/token'
data = {
    'grant_type': 'client_credentials',
    'client_id': 'your_client_id',
    'client_secret': 'your_client_secret'
}
response = requests.post(token_url, data=data)
access_token = response.json()['access_token']
# Use token
api_url = 'http://example.com/api/endpoint'
headers = {'Authorization': f'Bearer {access_token}'}
response = requests.get(api_url, headers=headers)
print(response.status_code)
print(response.json())