How to Scrape All Chinese Stock Data with Python: A Step‑by‑Step Guide
This tutorial explains how to collect the names and trading information of every stock listed on the Shanghai and Shenzhen exchanges using Python 3.5, requests, BeautifulSoup and regular expressions, then store the results in a file through a clear three‑step process of fetching the stock list, retrieving each stock's page, parsing the HTML and writing the data.
Function Overview
Goal: obtain the names and trading information of all stocks listed on the Shanghai and Shenzhen stock exchanges and save the results to a file. Technology stack: Python 3.5, requests, BeautifulSoup, regular expressions.
Explanation
Choose websites where stock information is static in the HTML source, not generated by JavaScript, and without robots.txt restrictions. Baidu Stock pages that render data via JavaScript are unsuitable, while pages whose data appears directly in the HTML are suitable. The Eastmoney stock list page provides a complete list of stock codes.
Principle Analysis
Steps:
Step 1: fetch the stock list from Eastmoney.
Step 2: construct URLs for each stock on Baidu Stock and retrieve their pages.
Step 3: parse the HTML to extract stock information and store it in a dictionary.
Step 4: write the dictionary data to a file.
Code Implementation
Core functions are shown below.
def getHTMLText(url):
try:
r = requests.get(url)
r.raise_for_status()
r.encoding = r.apparent_encoding
return r.text
except:
return ""def getStockList(lst, stockURL):
html = getHTMLText(stockURL)
soup = BeautifulSoup(html, 'html.parser')
a = soup.find_all('a')
for i in a:
try:
href = i.attrs['href']
lst.append(re.findall(r"[s][hz]\d{6}", href)[0])
except:
continuedef getStockInfo(lst, stockURL, fpath):
for stock in lst:
url = stockURL + stock + ".html"
html = getHTMLText(url)
if html == "":
continue
infoDict = {}
soup = BeautifulSoup(html, 'html.parser')
stockInfo = soup.find('div', attrs={'class': 'stock-bets'})
name = stockInfo.find_all('div', attrs={'class': 'bets-name'})[0].text.split()[0]
infoDict.update({'股票名称': name})
keyList = stockInfo.find_all('dt')
valueList = stockInfo.find_all('dd')
for i in range(len(keyList)):
key = keyList[i].text
val = valueList[i].text
infoDict[key] = val
with open(fpath, 'a', encoding='utf-8') as f:
f.write(str(infoDict) + '
')def main():
stock_list_url = 'http://quote.eastmoney.com/stocklist.html'
stock_info_url = 'https://gupiao.baidu.com/stock/'
output_file = 'D:/BaiduStockInfo.txt'
slist = []
getStockList(slist, stock_list_url)
getStockInfo(slist, stock_info_url, output_file)
if __name__ == '__main__':
main()The print statements in the original script display crawling progress. After execution, a file named BaiduStockInfo.txt will be created on the D: drive containing the collected stock information.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
