When to Use requests vs grequests: Sync, Threaded, and Async HTTP Calls in Python

Overview

Both requests and grequests are Python libraries for making HTTP calls, but they differ in execution model: requests performs blocking, single‑threaded calls, while grequests builds on requests and gevent to provide asynchronous, non‑blocking requests using coroutines.

Single‑threaded Synchronous Requests (requests)

import requests
urls = ["https://example.com/page1", "https://example.com/page2", "https://example.com/page3"]
# Single‑threaded synchronous requests
responses = [requests.get(url) for url in urls]
# Process responses
for response in responses:
    print(response.text)

Multi‑threaded Synchronous Requests (requests + threading)

import requests
import threading

def fetch_page(url):
    response = requests.get(url)
    return response.text

urls = ["https://example.com/page1", "https://example.com/page2", "https://example.com/page3"]
threads = []
# Start a thread for each URL
for url in urls:
    thread = threading.Thread(target=fetch_page, args=(url,))
    threads.append(thread)
    thread.start()
# Wait for all threads to finish
for thread in threads:
    thread.join()
# Collect results (example placeholder)
results = [result for result in thread_results]

Asynchronous Concurrent Requests (grequests)

import grequests
urls = ["https://example.com/page1", "https://example.com/page2", "https://example.com/page3"]
# Create a generator of unsent requests
rs = (grequests.get(url) for url in urls)
# Send them concurrently and collect responses
responses = grequests.map(rs)
# Process responses
for response in responses:
    print(response.text)

Timeout Control with grequests

import grequests
# Set a 5‑second timeout for each request
urls = [("https://example.com/page{}".format(i), {'timeout': 5}) for i in range(1, 4)]
rs = (grequests.get(url, **kwargs) for url, kwargs in urls)
responses = grequests.map(rs)
for response in responses:
    if response is not None:
        print(response.text)
    else:
        print(f"Request timed out for {response.request.url}")

Exception Handling in Asynchronous Requests (grequests)

import grequests
urls = ["https://example.com/page1", "https://example.com/page2", "https://example.com/page3"]
rs = [grequests.get(url) for url in urls]

def handle_exception(request, exception):
    print(f"Error on URL {request.url}: {exception}")

# Map with a custom exception handler and a concurrency limit
responses = grequests.map(rs, size=10, exception_handler=handle_exception)
for response in responses:
    if response is not None:
        print(f"{response.request.url} - Response: {response.text}")

In the final example, grequests.map() sends multiple requests concurrently; if any request raises an exception, the user‑defined handle_exception function logs the error without crashing the whole program, and the size parameter limits the maximum number of simultaneous connections to avoid overloading the target server.