Step‑by‑Step Python Guide to Scrape Fund Data and Export to CSV

Hello, I'm a Python enthusiast. A follower asked for fund information, so I share how to scrape fund data from a public website using Python.

1. Introduction

The target website is a fund official site. The fund code column lists numeric identifiers; clicking any code opens a detail page whose URL follows a predictable pattern based on the code.

The site does not encrypt its data; all information appears directly in the HTML source.

By capturing network requests, we see that the only variable parameter is pi, which represents the page number. Constructing a request with this parameter retrieves the desired data.

2. Code Implementation

Fetching fund IDs

response = requests.get(url, headers=headers, params=params, verify=False)
pattern = re.compile(r'.*?"(?P<items>.*?)".*?', re.S)
result = re.finditer(pattern, response.text)
ids = []
for item in result:
    # print(item.group('items'))
    gp_id = item.group('items').split(',')[0]

The above code extracts fund IDs from the list page.

Retrieving fund details

response = requests.get(url, headers=headers)
response.encoding = response.apparent_encoding
selectors = etree.HTML(response.text)
danweijingzhi1 = selectors.xpath('//dl[@class="dataItem02"]/dd[1]/span[1]/text()')[0]
danweijingzhi2 = selectors.xpath('//dl[@class="dataItem02"]/dd[1]/span[2]/text()')[0]
leijijingzhi = selectors.xpath('//dl[@class="dataItem03"]/dd[1]/span/text()')[0]
lst = selectors.xpath('//div[@class="infoOfFund"]/table//text()')

After extracting the required fields, the data is cleaned and saved to a CSV file.

3. Conclusion

This tutorial demonstrates a straightforward Python web‑scraping workflow for obtaining fund information, highlighting potential pitfalls and encouraging readers to adapt the parameters for other fund categories.