Boost Stock Data Download Speed with Python Threading & Queue

Using Python's threading and Queue modules, you can implement a producer‑consumer multithreaded model to accelerate the download of daily adjusted stock market data via the Tushare API.

Previously, serial downloading of adjusted data was slow and sometimes required retries; each stock requires a download step and a storage step, both of which can be parallelized.

Typical workflow

The usual pattern with queue and threading follows these steps:

Relevant Tushare interfaces

1. Stock list information

Purpose: Get basic information of listed companies on Shanghai and Shenzhen.

Method:

Result:

2. Daily adjusted market data

Purpose: Provide all historical data since listing, defaulting to forward‑adjusted.

Method:

Result:

Implementation

Producer thread – reads market data.

Consumer thread – stores data locally.

Main thread – coordinates threads.

Result: What originally took 2–3 hours now finishes in under one hour. Adding more threads yields diminishing returns because the Sina API may return HTTP 503 under high concurrency; using proxy pools and varying time windows can help.