Understanding HTTP Protocol and Using Python requests Library

In Python network programming, the HTTP protocol plays a crucial role as the foundation for communication between web applications. The requests library is the most popular HTTP client library in Python, making interaction with HTTP servers exceptionally simple. This chapter explores HTTP basics and demonstrates how to use requests for various HTTP requests.

1. HTTP Protocol Basics

HTTP (Hypertext Transfer Protocol) is a request‑response protocol used to transfer data between a web server and a client (usually a browser). The client sends a request, and the server returns a response.

1.1 HTTP Request

An HTTP request consists of the following main parts:

Request Line : specifies the method, URI, and HTTP version.

Method : e.g., GET, POST, PUT, DELETE —each representing a different operation.

URI (Uniform Resource Identifier) : identifies the resource on the server, e.g., /index.html or /api/users.

HTTP Version : e.g., HTTP/1.1 or HTTP/2.

Request Headers : additional information sent by the client, such as User-Agent , Content-Type , and Accept . User-Agent: identifies the client type and version. Content-Type: specifies the media type of the request body. Accept: indicates the response content types the client can handle.

Request Body : contains data sent to the server, typically used with POST or PUT methods.

1.2 HTTP Response

An HTTP response includes the following main parts:

Status Line : contains the HTTP version, status code, and status text.

HTTP Version : e.g., HTTP/1.1.

Status Code : three‑digit code indicating the result, e.g., 200 OK, 404 Not Found, 500 Internal Server Error.

Status Text : short description of the status code, e.g., OK, Not Found, Internal Server Error.

Response Headers : additional information from the server, such as Content-Type , Content-Length , and Server . Content-Type: type of data in the response body. Content-Length: length of the response body. Server: identifies the server software and version.

Response Body : the actual data returned by the server, often HTML, JSON, or other formats.

2. Using the requests Library

2.1 Install the requests library

pip install requests

2.2 Send a GET request

import requests

url = "https://www.example.com"

response = requests.get(url)

print(f"Status code: {response.status_code}")
print(f"Headers: {response.headers}")
print(f"Content: {response.text}")

2.3 Send a POST request

import requests

url = "https://httpbin.org/post"  # simulate POST with httpbin.org

data = {"key1": "value1", "key2": "value2"}

response = requests.post(url, data=data)

print(f"Status code: {response.status_code}")
print(f"Content: {response.text}")

2.4 Send a GET request with parameters

import requests

url = "https://httpbin.org/get"

params = {"key1": "value1", "key2": "value2"}

response = requests.get(url, params=params)

print(f"Status code: {response.status_code}")
print(f"Content: {response.text}")

2.5 Set custom request headers

import requests

url = "https://www.example.com"

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"}

response = requests.get(url, headers=headers)

print(f"Status code: {response.status_code}")
print(f"Headers: {response.headers}")
print(f"Content: {response.text}")

2.6 Process JSON responses

import requests

url = "https://httpbin.org/ip"

response = requests.get(url)

if response.status_code == 200:
    data = response.json()
    print(f"IP address: {data['origin']}")
else:
    print(f"Request failed: {response.status_code}")

2.7 Set a timeout

import requests

url = "https://www.example.com"

try:
    response = requests.get(url, timeout=5)  # 5‑second timeout
    print(f"Status code: {response.status_code}")
    print(f"Content: {response.text}")
except requests.exceptions.Timeout:
    print("Request timed out")

2.8 Exception handling

The requests library raises exceptions such as requests.exceptions.ConnectionError (connection error) and requests.exceptions.HTTPError (HTTP error). Use try...except blocks to handle them.

import requests

url = "https://www.example.com"

try:
    response = requests.get(url)
    response.raise_for_status()  # raise HTTPError for bad status codes
    print(f"Status code: {response.status_code}")
    print(f"Content: {response.text}")
except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")

3. Summary

This chapter covered the basics of the HTTP protocol and demonstrated how to use the requests library for sending GET, POST, PUT, DELETE requests, setting headers, handling JSON responses, configuring timeouts, and managing exceptions. Mastering these concepts enables efficient network programming and the development of robust web applications.