How to Scrape Fund Data with Python: A Step‑by‑Step Guide

Introduction

Hello, I am a Python enthusiast sharing a practical example of using a web crawler to obtain fund data from a public website. The target site does not encrypt its data, so the information is directly visible in the page source.

Data Acquisition

The goal is to scrape a fund website that lists fund codes. By clicking any code you can reach a detail page whose URL follows a predictable pattern based on the code.

Using the browser's network inspector you can see that the request only varies by a single parameter pi, which corresponds to the page number. Constructing a request with this parameter allows you to retrieve the data programmatically.

Implementation

Below are the key code snippets used to fetch the stock IDs and then retrieve detailed fund information.

Fetching Stock IDs

response = requests.get(url, headers=headers, params=params, verify=False)
pattern = re.compile(r'.*?"(?P<items>.*?)".*?', re.S)
result = re.finditer(pattern, response.text)
ids = []
for item in result:
    # print(item.group('items'))
    gp_id = item.group('items').split(',')[0]

Fetching Fund Details

response = requests.get(url, headers=headers)
response.encoding = response.apparent_encoding
selectors = etree.HTML(response.text)
unit_value1 = selectors.xpath('//dl[@class="dataItem02"]/dd[1]/span[1]/text()')[0]
unit_value2 = selectors.xpath('//dl[@class="dataItem02"]/dd[1]/span[2]/text()')[0]
total_value = selectors.xpath('//dl[@class="dataItem03"]/dd[1]/span/text()')[0]
info_list = selectors.xpath('//div[@class="infoOfFund"]/table//text()')

The extracted strings are then cleaned and written to a csv file, as shown in the following screenshot.

Conclusion

This tutorial demonstrates that scraping fund data with Python is straightforward when the website does not employ obfuscation. After obtaining the data you can perform further statistical analysis or visualisation. The same approach can be adapted to other fund categories by adjusting the request parameters.