How to Download All Files with a Python Web Scraper: Step-by-Step Guide

1. Introduction

Hello, I'm Pipi. A few days ago in the Python Diamond group a member asked a question about Python web crawling, which I share here.

He wanted to retrieve all files, but in practice could only fetch a single file.

2. Implementation

Classmate Ning provided a solution idea, as shown in the figure:

Later, Teacher Yuliang offered another method, illustrated below:

A direct one‑step solution is shown here:

Below is the complete code:

import requests
import time
from lxml import html
from lxml import etree

headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 '
    'Safari/537.36'
}

def get_href_links(url):
    response = requests.get(url, headers=headers)
    page_content = response.content
    dom_tree = html.fromstring(page_content)
    href_links = dom_tree.xpath('//a/@href')
    return href_links

url = "https://mp.weixin.qq.com/s/BANHI5apQzlpeTTdLZAIvg"
urls = set(get_href_links(url)[1:-5])
mp3_d_url = 'https://res.wx.qq.com/voice/getvoice?mediaid={}'
for url in urls:
    response = requests.get(url, headers=headers)
    html_text = response.text
    selector = etree.HTML(html_text)
    voice_encode_fileid = selector.xpath('//mpvoice/@voice_encode_fileid')[0]
    name = selector.xpath('//mpvoice/@name')[0]
    d_url = mp3_d_url.format(voice_encode_fileid)
    response = requests.get(d_url)
    if response.status_code == 200:
        with open(name, 'wb') as f:
            f.write(response.content)
        print(f'{name} 下载成功！')
    else:
        print(f'{name} 下载失败！')
    time.sleep(1)  # 设置请求间隔时间为1秒，避免被封IP

The script successfully solved the fan's problem.

3. Conclusion

This article addresses a Python web‑crawling issue, providing analysis and a complete implementation to help solve it.

Tip: When posting large files in a group, consider data masking and share only demo (small) files, include reproducible code snippets, and attach error screenshots if needed.