Implement Robust HTTP Retry with urllib3 & Requests in Python

When handling HTTP requests, many errors can occur, ranging from simple timeouts to 429 Too Many Requests. Python's standard library provides a native solution called urllib3 that lets you define which HTTP methods to check, which status codes are expected, and how long to wait before retrying.

Retry on Failure

Network connections may be lost or congested, and servers can fail. To build a truly robust program, you need to consider these potential faults and devise a retry strategy.

Adding a retry strategy to an HTTP client is straightforward: create an HTTPAdapter and pass your strategy to the adapter.

The default Retry class provides a standard configuration but is highly configurable. Below are common parameters.

Total retry attempts : total = 3 If the number of failed requests or redirects exceeds this value, the client raises urllib3.exceptions.MaxRetryError. Typically, a value less than 10 is used, with 3 attempts being sufficient for most cases.

Status codes to retry : status_forcelist = [413, 429, 503] You may want to retry on common server errors (500, 502, 503, 504) because servers and reverse proxies do not always follow the HTTP spec. Always retry on 429 (rate‑limit) as the urllib library will back off automatically.

HTTP methods to retry :

method_whitelist = ["HEAD", "GET", "PUT", "DELETE", "OPTIONS", "TRACE"]

By default, all methods except POST are included, since POST can cause new inserts.

Backoff factor : backoff_factor = 0 This factor controls the pause between failed requests using the formula:

{backoff factor} * (2 ** ({number of total retries} - 1))

Examples:

1 second : subsequent delays will be 0.5, 1, 2, 4, 8, 16, 32, 64, 128, 256 seconds.

2 seconds : subsequent delays will be 1, 2, 4, 8, 16, 32, 64, 128, 256, 512 seconds.

10 seconds : subsequent delays will be 5, 10, 20, 40, 80, 160, 320, 640, 1280, 2560 seconds.

The default backoff factor is 0, meaning no exponential backoff; new attempts happen immediately. Set this parameter to avoid overwhelming the server.

For more details, refer to the full documentation of the Retry module.

Practical Example

Assume the API you consume allows 100 requests per minute. Exceeding this limit returns a 429 status code. The following code applies a retry strategy to handle this situation:

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
import requests

retries = Retry(
    total=5,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504],
    method_whitelist=["HEAD", "GET", "PUT", "DELETE", "OPTIONS", "TRACE"]
)

session = requests.Session()
session.mount('https://', HTTPAdapter(max_retries=retries))

response = session.get('https://api.example.com/data')
print(response.status_code)

This code will retry the request each minute when the rate limit is exceeded.

Summary

As demonstrated, Retry is a powerful tool; when used correctly, it is essential for building robust API consumption solutions.