Using Python Requests Library for Common HTTP Operations and API Automation

GET Request Example

Requests is the most popular HTTP library in Python; the following code shows a simple GET request to retrieve data from an API.

import requests
url = "https://api.example.com/data"
response = requests.get(url)
if response.status_code == 200:
    print("Data received:", response.json())
else:
    print("Request failed with status code:", response.status_code)

POST Request to Submit Form Data

Submitting data such as login credentials typically uses a POST request.

import requests
url = "https://api.example.com/login"
payload = {"username": "[email protected]", "password": "securepass"}
response = requests.post(url, data=payload)
if response.status_code == 200:
    print("Login successful!")
else:
    print("Login failed with status code:", response.status_code)

Using Session to Keep a Logged‑In State

A Session object preserves cookies across requests, allowing subsequent calls to authenticated endpoints.

import requests
s = requests.Session()
login_url = "https://api.example.com/login"
login_payload = {"username": "[email protected]", "password": "securepass"}
# Login
s.post(login_url, data=login_payload)
# Access a protected page using the same session
protected_url = "https://api.example.com/profile"
response = s.get(protected_url)
if response.status_code == 200:
    print("Profile data received:", response.json())

Exception Handling and Retry Logic

Adding error handling and retry attempts makes scripts more robust.

import requests
from requests.exceptions import RequestException
import time

def fetch_data_with_retry(url, retries=3):
    for attempt in range(retries):
        try:
            response = requests.get(url)
            response.raise_for_status()
            return response.json()
        except RequestException as e:
            print(f"Attempt {attempt + 1} failed. Retrying...")
            time.sleep(2 ** attempt)
    print("Failed to fetch data after all attempts.")
    return None

data = fetch_data_with_retry("https://api.example.com/data")

Using Params for Query Parameters

When adding query strings to a URL, the params argument simplifies the process.

import requests
url = "https://api.example.com/search"
params = {"query": "python", "limit": 10}
response = requests.get(url, params=params)
results = response.json()
print("Search results:", results)

Automated API Test Report

Combining unittest (or pytest) with Requests enables automatic test case execution and reporting.

import requests
import unittest

class TestAPI(unittest.TestCase):
    def test_get_data(self):
        url = "https://api.example.com/data"
        response = requests.get(url)
        self.assertEqual(response.status_code, 200)

if __name__ == '__main__':
    unittest.main(testRunner=unittest.TextTestRunner())

Validate Responses with JSON Schema

Ensuring the API response matches an expected structure can be done with the jsonschema library.

import requests
import jsonschema
from jsonschema import validate
from jsonschema.exceptions import ValidationError

schema = {
    "type": "object",
    "properties": {
        "id": {"type": "integer"},
        "name": {"type": "string"}
    },
    "required": ["id", "name"]
}
url = "https://api.example.com/user/1"
response = requests.get(url)
try:
    validate(instance=response.json(), schema=schema)
    print("Response data is valid according to schema.")
except ValidationError as e:
    print("Validation error:", e.message)

File Upload

Uploading files such as images or documents to an API is straightforward with the files parameter.

import requests
url = "https://api.example.com/upload"
files = {'file': open('example.txt', 'rb')}
response = requests.post(url, files=files)
if response.status_code == 200:
    print("File uploaded successfully.")
else:
    print("Upload failed with status code:", response.status_code)

Multithreading/Asynchronous Requests for Higher Throughput

Using the concurrent.futures module to run multiple requests in parallel speeds up data collection.

import requests
from concurrent.futures import ThreadPoolExecutor

urls = ["https://api.example.com/data1", "https://api.example.com/data2"]

def fetch_url(url):
    response = requests.get(url)
    return url, response.status_code

with ThreadPoolExecutor(max_workers=5) as executor:
    futures = {executor.submit(fetch_url, url) for url in urls}
    for future in concurrent.futures.as_completed(futures):
        url, status = future.result()
        print(f"{url}: Status Code {status}")

Downloading Large Files with requests‑toolbelt

For large downloads, requests-toolbelt provides a streaming iterator that avoids loading the entire file into memory.

from requests_toolbelt.downloadutils import StreamingIterator
import requests
url = "https://example.com/largefile.zip"
response = requests.get(url, stream=True)
stream = StreamingIterator(response.iter_content(chunk_size=1024*1024),  # 1 MB chunks
                          headers={'Accept-Encoding': None})  # disable automatic decompression
with open('largefile.zip', 'wb') as f:
    for chunk in stream:
        f.write(chunk)
print("Download complete.")

These examples cover common needs in API automation, including basic HTTP requests, error handling, data validation, file operations, and performance optimizations; mastering them can significantly improve the efficiency and quality of API testing and data extraction workflows.