How to Scrape Toutiao Street‑Photo Images with Python: A Step‑by‑Step Guide

Introduction

In this tutorial we demonstrate how to use Python to scrape street‑photo images from Toutiao by analyzing the Ajax requests that deliver the image data and then programmatically downloading the pictures.

Analyzing the Ajax request

Opening the search page and inspecting the Network tab reveals requests whose URLs start with search?keyword=%. The JSON response contains fields img_small_url and img_url that hold the image links.

Only the page_num parameter changes between requests, so it can be used for pagination.

Key request parameters

The GET request includes the following parameters: keyword, pd, dvpf, aid, page_num, search_json, rawJSON, search_id. All remain constant except page_num.

Implementation

We define three functions:

def get_page(page_num):
    global headers
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36'
    }
    params = {
        'keyword': urllib.parse.unquote('%E8%A1%97%E6%8B%8D'),
        'pd': 'atlas',
        'dvpf': 'pc',
        'aid': 4916,
        'page_num': page_num,
        'search_json': '%7B%22from_search_id%22%3A%22202106100003510102121720341003A4ED%22%2C%22origin_keyword%22%3A%22%E8%A1%97%E6%8B%8D%22%2C%22image_keyword%22%3A%22%E8%A1%97%E6%8B%8D%22%7D',
        'rawJSON': 1,
        'search_id': '202106100004290101500200495C05B763'
    }
    url = 'https://so.toutiao.com/search?' + urlencode(params)
    try:
        response = requests.get(url, headers=headers, params=params)
        if response.status_code == 200:
            return response.json()
    except requests.ConnectionError:
        return None

def get_images(json):
    images = json.get('rawData').get('data')
    for image in images:
        link = image.get('img_url')
        yield link

def saving_img(link):
    global name
    print(f'-------Downloading image {name}')
    data = requests.get(link, headers=headers).content
    with open(f'image1/{name}.jpg', 'wb') as f:
        f.write(data)
        name += 1

The main routine iterates over a range of page_num values, calls get_page, extracts image URLs with get_images, and saves each picture with saving_img.

def main(page_num):
    json = get_page(page_num)
    for link in get_images(json):
        saving_img(link)

if __name__ == '__main__':
    for i in range(0, 2):
        main(i)

Result

Running the script downloads the street‑photo images to the local image1 folder; in the example two pages yield about 80 pictures.

Conclusion

The tutorial shows a complete end‑to‑end Python crawler for Toutiao street‑photo images and suggests further improvements such as using a multiprocessing pool for faster downloads.