Using pytest‑xdist for Parallel and Distributed Testing in Python

Introduction pytest‑xdist is a powerful plugin that enables parallel execution of test cases across multiple processes, significantly speeding up test runs, especially for large projects.

1. Install pytest‑xdist Install the plugin via pip:

pip install pytest-xdist

2. Basic usage

2.1 Using the -n option Specify the number of parallel processes with the -n flag, e.g., to start four processes:

pytest -n 4

2.2 Using -n auto Let pytest automatically detect the number of CPU cores and launch the appropriate number of processes:

pytest -n auto

3. Distributed testing

3.1 Starting test nodes On one or more remote machines, start a test node:

pytest --workerinput nodeid=node1 --workerinput hostname=remotehost1

3.2 Running distributed tests From the master node, launch tests and specify remote nodes:

pytest -n 4 --tx ssh=remotehost1 --tx ssh=remotehost2

This runs four processes across the local and remote hosts.

4. Concrete example

Assume a test file test_example.py containing several test cases:

import time

def test_add():
    assert 1 + 1 == 2
    time.sleep(1)

def test_subtract():
    assert 2 - 1 == 1
    time.sleep(1)

def test_multiply():
    assert 2 * 3 == 6
    time.sleep(1)

def test_divide():
    assert 6 / 3 == 2
    time.sleep(1)

def test_power():
    assert 2 ** 3 == 8
    time.sleep(1)

4.2 Parallel execution Run the tests with four parallel processes:

pytest -n 4 test_example.py

4.3 Sample output The output shows each test case executing in parallel:

============================= test session starts =============================
platform linux -- Python 3.8.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /path/to/tests
plugins: xdist-2.1.0
collected 5 items
test_example.py [100%] (4 processes)
=============================== 5 passed in 1.06s ===============================

5. Configuration file You can set default options in pytest.ini (or .pylintrc ) to always start four processes:

# ini
[pytest]
addopts = -n 4

6. Advanced usage

6.1 Custom process names Assign names to processes for easier tracking:

pytest -n 4 --tx "popen//id=node1" --tx "popen//id=node2" --tx "popen//id=node3" --tx "popen//id=node4"

6.2 Specifying ports When launching remote nodes, you can specify a port:

pytest --workerinput nodeid=node1 --workerinput hostname=remotehost1 --workerinput port=5555

6.3 Using SSH tunnels Start remote nodes through an SSH tunnel using the -k option:

pytest --workerinput nodeid=node1 --workerinput hostname=remotehost1 --workerinput port=5555 --workerinput tunnel=ssh

7. Summary The article covered installation, basic and advanced usage of pytest‑xdist, including parallel execution with -n , distributed testing across machines, example test code, configuration via pytest.ini , and advanced options such as custom process identifiers, port specification, and SSH tunneling.