How to Scrape All Chinese Stock Data with Python: A Step‑by‑Step Guide

Function Overview

Goal: obtain the names and trading information of all stocks listed on the Shanghai and Shenzhen stock exchanges and save the results to a file. Technology stack: Python 3.5, requests, BeautifulSoup, regular expressions.

Explanation

Choose websites where stock information is static in the HTML source, not generated by JavaScript, and without robots.txt restrictions. Baidu Stock pages that render data via JavaScript are unsuitable, while pages whose data appears directly in the HTML are suitable. The Eastmoney stock list page provides a complete list of stock codes.

Principle Analysis

Steps:

Step 1: fetch the stock list from Eastmoney.

Step 2: construct URLs for each stock on Baidu Stock and retrieve their pages.

Step 3: parse the HTML to extract stock information and store it in a dictionary.

Step 4: write the dictionary data to a file.

Code Implementation

Core functions are shown below.

def getHTMLText(url):
    try:
        r = requests.get(url)
        r.raise_for_status()
        r.encoding = r.apparent_encoding
        return r.text
    except:
        return ""

def getStockList(lst, stockURL):
    html = getHTMLText(stockURL)
    soup = BeautifulSoup(html, 'html.parser')
    a = soup.find_all('a')
    for i in a:
        try:
            href = i.attrs['href']
            lst.append(re.findall(r"[s][hz]\d{6}", href)[0])
        except:
            continue

def getStockInfo(lst, stockURL, fpath):
    for stock in lst:
        url = stockURL + stock + ".html"
        html = getHTMLText(url)
        if html == "":
            continue
        infoDict = {}
        soup = BeautifulSoup(html, 'html.parser')
        stockInfo = soup.find('div', attrs={'class': 'stock-bets'})
        name = stockInfo.find_all('div', attrs={'class': 'bets-name'})[0].text.split()[0]
        infoDict.update({'股票名称': name})
        keyList = stockInfo.find_all('dt')
        valueList = stockInfo.find_all('dd')
        for i in range(len(keyList)):
            key = keyList[i].text
            val = valueList[i].text
            infoDict[key] = val
        with open(fpath, 'a', encoding='utf-8') as f:
            f.write(str(infoDict) + '
')

def main():
    stock_list_url = 'http://quote.eastmoney.com/stocklist.html'
    stock_info_url = 'https://gupiao.baidu.com/stock/'
    output_file = 'D:/BaiduStockInfo.txt'
    slist = []
    getStockList(slist, stock_list_url)
    getStockInfo(slist, stock_info_url, output_file)

if __name__ == '__main__':
    main()

The print statements in the original script display crawling progress. After execution, a file named BaiduStockInfo.txt will be created on the D: drive containing the collected stock information.