Backend Development 13 min read

Master HTTP/HTTPS Testing with Python httpx and curl: A Practical Guide

This guide explains how to use Python's httpx library and the curl command‑line tool to perform comprehensive HTTP/HTTPS testing—including basic requests, chunked transfers, HTTP/2, SSL/TLS configuration, and dynamic DNS resolution—complete with code examples and setup instructions.

Tencent Architect

Feb 28, 2023

Master HTTP/HTTPS Testing with Python httpx and curl: A Practical Guide

1. Preparation

1.1 httpx

The httpx library supports Python 3 and can be installed via pip. It supports HTTP/2 and provides a CLI client.

python -m pip install httpx</code>
<code># Enable HTTP/2</code>
<code>python -m pip install "httpx[http2]"</code>
<code># Install CLI client</code>
<code>python -m pip install "httpx[cli]"

Python usage example:

>> import httpx
>>> r = httpx.get('https://www.example.org/')
>>> r
<Response [200 OK]>
>>> r.status_code
200
>>> r.headers['content-type']
'text/html; charset=UTF-8'
>>> r.text
'<!doctype html>
<html>
<head>
<title>Example Domain</title>...'

1.2 curl

Upgrade curl to version 7.68.0 or later. To enable HTTP/2, install nghttp2.

# Clone and build nghttp2
git clone https://github.com/tatsuhiro-t/nghttp2.git
cd nghttp2-master
autoreconf -i
automake
autoconf
./configure
make && make install
echo '/usr/local/lib' > /etc/ld.so.conf.d/local.conf
ldconfig

# Upgrade curl
yum install build-dep curl
wget http://curl.haxx.se/download/curl-7.68.0.zip
unzip curl-7.68.0.zip
cd curl-7.68.0
./configure --with-nghttp2=/usr/local --with-ssl
make && make install
ldconfig

During ./configure, you can see that HTTP/2 is enabled.

2. Start

2.1 Basic Requests

2.1.1 httpx.Client

For flexible code, use httpx.Client or httpx.AsyncClient and call client.request() instead of the shortcut functions.

with httpx.Client() as client:
    client.request(
        method=method,
        url=req_url,
        headers=req_headers,
        content=content
    )

2.1.2 curl

curl <url> -X <method> -d <data> -H <header>
# For large POST/PUT files, use -F 'file=@<filename>'
# For HEAD requests, use -I instead of -X HEAD

2.2 Chunked Transfer

Chunked transfer encoding can be simulated by passing a byte iterator to httpx.request().

if req_chunked:
    _content = content
    _middle = _content.__len__() // 2
    async def content():
        yield _content[:_middle]
        yield _content[_middle:]
else:
    pass

async with httpx.AsyncClient(http2=http2) as client:
    task = asyncio.create_task(
        client.request(
            method=method,
            url=req_url,
            headers=req_headers,
            content=content() if callable(content) else content
        )
    )
    try:
        await task
    except asyncio.CancelledError:
        pass

2.3 HTTP/2

httpx supports HTTP/2 via the http2=True parameter; curl uses --http2.

with httpx.Client(http2=True) as client:
    client.request(method=method, url=req_url, headers=req_headers, content=content)

curl <url> --http2

2.3.1 Multiplexed Requests

Use httpx.AsyncClient to send concurrent requests over a single connection.

async with httpx.AsyncClient(http2=http2) as client:
    for i in range(req_num):
        task_list = []
        for m in range(multiplexing):
            task = asyncio.create_task(
                client.request(method=method, url=req_url, headers=req_headers, content=content)
            )
            task_list.append(task)
        await asyncio.wait(task_list)
        for t in task_list:
            try:
                await t
            except _err_type:
                pass
            if t.exception():
                logger.warning(t.exception())
            else:
                pass
            response_list.append(t.result())

2.4 SSL/TLS

Python's built‑in ssl module can configure cipher suites and protocol versions. curl provides --ciphers, --tls-max, and --tlsv1.x options.

2.4.1 Ciphers

import httpx
import ssl

ssl_ctx = ssl.create_default_context()
ssl_ctx.verify_flags = ssl.VerifyFlags.VERIFY_DEFAULT
ssl_ctx.check_hostname = False
ssl_ctx.verify_mode = ssl.VerifyMode.CERT_NONE
ssl_ctx.set_ciphers("AES128-GCM-SHA256")

url = "https://<vip>:<vport>/"
rsp = httpx.get(url, verify=ssl_ctx)
"""
>>> rsp.status_code
200
>>> rsp.headers
Headers({'date': 'Fri, 01 Jul 2022 08:25:55 GMT', 'content-type': 'application/octet-stream', 'transfer-encoding': 'chunked', ...})
"""

# Query supported ciphers
ssl_ctx.get_ciphers()

curl <https url> -k --cipher ECDHE-RSA-AES256-GCM-SHA384

List system‑supported ciphers: openssl ciphers 2.4.2 Specifying TLS Version

import httpx
import ssl

ssl_ctx = ssl.create_default_context()
ssl_ctx.verify_flags = ssl.VerifyFlags.VERIFY_DEFAULT
ssl_ctx.check_hostname = False
ssl_ctx.verify_mode = ssl.VerifyMode.CERT_NONE

# Set TLS version (example: TLS 1.1)
ssl_ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1_1)
print(ssl_ctx.protocol)  # <_SSLMethod.PROTOCOL_TLSv1_1: 4>

url = "https://<vip>:<vport>/"
rsp = httpx.get(url, verify=ssl_ctx)
"""
>>> rsp.status_code
200
>>> rsp.headers
Headers({'date': 'Mon, 04 Jul 2022 06:28:08 GMT', 'content-type': 'application/octet-stream', ...})
"""

# curl example (TLS 1.2)
curl <https url> -k --tlsv1.2 --tls-max 1.2

2.5 Resolve / SNI

SNI allows a server to host multiple certificates. To avoid editing /etc/hosts, dynamically resolve domain names at runtime.

Python example – override socket.getaddrinfo:

import ipaddress
import socket
from loguru import logger

class DNS(object):
    DNS_CACHE = {}
    def __init__(self):
        self.socket_get_address_info = socket.getaddrinfo
        socket.getaddrinfo = self.custom_get_address_info
    def add_custom_dns(self, domain: str, port: int, ip: str):
        key = (domain.encode("utf-8"), port)
        if ipaddress.ip_address(ip).version == 4:
            value = (socket.AddressFamily.AF_INET, socket.SocketKind.SOCK_STREAM, socket.IPPROTO_TCP, '', (ip, port))
        else:
            value = (socket.AddressFamily.AF_INET6, socket.SocketKind.SOCK_STREAM, socket.IPPROTO_TCP, '', (ip, port, 0, 0))
        self.DNS_CACHE[key] = [value]
        logger.debug(f"DNS_Cache: {self.DNS_CACHE}")
    def custom_get_address_info(self, *args):
        logger.debug(f"Args: {args}")
        try:
            if isinstance(args[0], str):
                key = (args[0].encode("utf8"), args[1])
            else:
                key = args[:2]
            return self.DNS_CACHE[key]
        except KeyError:
            return self.socket_get_address_info(*args)

curl example – use --resolve for a one‑time domain‑to‑IP mapping:

curl <url> --resolve <domain>:<port>:<ip>

3. References

https://www.python-httpx.org

pypi.org/project/httpx/

requests.readthedocs.io

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.

Python HTTP2 cURL TLS httpx

Written by

Tencent Architect

We share technical insights on storage, computing, and access, and explore industry-leading product technologies together.

0 followers

Reader feedback

How this landed with the community

Rate this article

Was this worth your time?

Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.