BestHub
Sign in
Backend Development 13 min read

Master HTTP/HTTPS Testing with Python httpx and curl: A Practical Guide

This guide explains how to use Python's httpx library and the curl command‑line tool to perform comprehensive HTTP/HTTPS testing—including basic requests, chunked transfers, HTTP/2, SSL/TLS configuration, and dynamic DNS resolution—complete with code examples and setup instructions.

Tencent Architect
Tencent Architect
Tencent Architect
Master HTTP/HTTPS Testing with Python httpx and curl: A Practical Guide

1. Preparation

1.1 httpx

The httpx library supports Python 3 and can be installed via pip. It supports HTTP/2 and provides a CLI client.

<code>python -m pip install httpx</code>
<code># Enable HTTP/2</code>
<code>python -m pip install "httpx[http2]"</code>
<code># Install CLI client</code>
<code>python -m pip install "httpx[cli]"</code>

Python usage example:

<code>>> import httpx
>>> r = httpx.get('https://www.example.org/')
>>> r
<Response [200 OK]>
>>> r.status_code
200
>>> r.headers['content-type']
'text/html; charset=UTF-8'
>>> r.text
'<!doctype html>\n<html>\n<head>\n<title>Example Domain</title>...'</code>

1.2 curl

Upgrade curl to version 7.68.0 or later. To enable HTTP/2, install nghttp2.

<code># Clone and build nghttp2
git clone https://github.com/tatsuhiro-t/nghttp2.git
cd nghttp2-master
autoreconf -i
automake
autoconf
./configure
make && make install
echo '/usr/local/lib' > /etc/ld.so.conf.d/local.conf
ldconfig</code>
<code># Upgrade curl
yum install build-dep curl
wget http://curl.haxx.se/download/curl-7.68.0.zip
unzip curl-7.68.0.zip
cd curl-7.68.0
./configure --with-nghttp2=/usr/local --with-ssl
make && make install
ldconfig</code>

During

./configure

, you can see that HTTP/2 is enabled.

2. Start

2.1 Basic Requests

2.1.1 httpx.Client

For flexible code, use

httpx.Client

or

httpx.AsyncClient

and call

client.request()

instead of the shortcut functions.

<code>with httpx.Client() as client:
    client.request(
        method=method,
        url=req_url,
        headers=req_headers,
        content=content
    )</code>

2.1.2 curl

<code>curl <url> -X <method> -d <data> -H <header>
# For large POST/PUT files, use -F 'file=@<filename>'
# For HEAD requests, use -I instead of -X HEAD</code>

2.2 Chunked Transfer

Chunked transfer encoding can be simulated by passing a byte iterator to

httpx.request()

.

<code>if req_chunked:
    _content = content
    _middle = _content.__len__() // 2
    async def content():
        yield _content[:_middle]
        yield _content[_middle:]
else:
    pass

async with httpx.AsyncClient(http2=http2) as client:
    task = asyncio.create_task(
        client.request(
            method=method,
            url=req_url,
            headers=req_headers,
            content=content() if callable(content) else content
        )
    )
    try:
        await task
    except asyncio.CancelledError:
        pass</code>

2.3 HTTP/2

httpx supports HTTP/2 via the

http2=True

parameter; curl uses

--http2

.

<code>with httpx.Client(http2=True) as client:
    client.request(method=method, url=req_url, headers=req_headers, content=content)</code>
<code>curl <url> --http2</code>

2.3.1 Multiplexed Requests

Use

httpx.AsyncClient

to send concurrent requests over a single connection.

<code>async with httpx.AsyncClient(http2=http2) as client:
    for i in range(req_num):
        task_list = []
        for m in range(multiplexing):
            task = asyncio.create_task(
                client.request(method=method, url=req_url, headers=req_headers, content=content)
            )
            task_list.append(task)
        await asyncio.wait(task_list)
        for t in task_list:
            try:
                await t
            except _err_type:
                pass
            if t.exception():
                logger.warning(t.exception())
            else:
                pass
            response_list.append(t.result())</code>

2.4 SSL/TLS

Python's built‑in

ssl

module can configure cipher suites and protocol versions. curl provides

--ciphers

,

--tls-max

, and

--tlsv1.x

options.

2.4.1 Ciphers

<code>import httpx
import ssl

ssl_ctx = ssl.create_default_context()
ssl_ctx.verify_flags = ssl.VerifyFlags.VERIFY_DEFAULT
ssl_ctx.check_hostname = False
ssl_ctx.verify_mode = ssl.VerifyMode.CERT_NONE
ssl_ctx.set_ciphers("AES128-GCM-SHA256")

url = "https://<vip>:<vport>/"
rsp = httpx.get(url, verify=ssl_ctx)
"""
>>> rsp.status_code
200
>>> rsp.headers
Headers({'date': 'Fri, 01 Jul 2022 08:25:55 GMT', 'content-type': 'application/octet-stream', 'transfer-encoding': 'chunked', ...})
"""

# Query supported ciphers
ssl_ctx.get_ciphers()
</code>
<code>curl <https url> -k --cipher ECDHE-RSA-AES256-GCM-SHA384</code>

List system‑supported ciphers:

<code>openssl ciphers</code>

2.4.2 Specifying TLS Version

<code>import httpx
import ssl

ssl_ctx = ssl.create_default_context()
ssl_ctx.verify_flags = ssl.VerifyFlags.VERIFY_DEFAULT
ssl_ctx.check_hostname = False
ssl_ctx.verify_mode = ssl.VerifyMode.CERT_NONE

# Set TLS version (example: TLS 1.1)
ssl_ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1_1)
print(ssl_ctx.protocol)  # <_SSLMethod.PROTOCOL_TLSv1_1: 4>

url = "https://<vip>:<vport>/"
rsp = httpx.get(url, verify=ssl_ctx)
"""
>>> rsp.status_code
200
>>> rsp.headers
Headers({'date': 'Mon, 04 Jul 2022 06:28:08 GMT', 'content-type': 'application/octet-stream', ...})
"""
</code>
<code># curl example (TLS 1.2)
curl <https url> -k --tlsv1.2 --tls-max 1.2</code>

2.5 Resolve / SNI

SNI allows a server to host multiple certificates. To avoid editing

/etc/hosts

, dynamically resolve domain names at runtime.

Python example – override

socket.getaddrinfo

:

<code>import ipaddress
import socket
from loguru import logger

class DNS(object):
    DNS_CACHE = {}
    def __init__(self):
        self.socket_get_address_info = socket.getaddrinfo
        socket.getaddrinfo = self.custom_get_address_info
    def add_custom_dns(self, domain: str, port: int, ip: str):
        key = (domain.encode("utf-8"), port)
        if ipaddress.ip_address(ip).version == 4:
            value = (socket.AddressFamily.AF_INET, socket.SocketKind.SOCK_STREAM, socket.IPPROTO_TCP, '', (ip, port))
        else:
            value = (socket.AddressFamily.AF_INET6, socket.SocketKind.SOCK_STREAM, socket.IPPROTO_TCP, '', (ip, port, 0, 0))
        self.DNS_CACHE[key] = [value]
        logger.debug(f"DNS_Cache: {self.DNS_CACHE}")
    def custom_get_address_info(self, *args):
        logger.debug(f"Args: {args}")
        try:
            if isinstance(args[0], str):
                key = (args[0].encode("utf8"), args[1])
            else:
                key = args[:2]
            return self.DNS_CACHE[key]
        except KeyError:
            return self.socket_get_address_info(*args)
</code>

curl example – use

--resolve

for a one‑time domain‑to‑IP mapping:

<code>curl <url> --resolve <domain>:<port>:<ip></code>

3. References

https://www.python-httpx.org

pypi.org/project/httpx/

requests.readthedocs.io

PythonHTTP2curlTLShttpx
Tencent Architect
Written by

Tencent Architect

We share technical insights on storage, computing, and access, and explore industry-leading product technologies together.

0 followers
Reader feedback

How this landed with the community

login Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.

Sign in to comment