Build a Python Stock Data Scraper with Requests and Pandas

Introduction

Hello, I'm PiPi. A few days ago a member asked a question about a Python web crawler in the Python Silver group, so I share the solution here.

Implementation

The initial idea came from ChatGPT, which can provide a basic approach but often requires manual adjustments such as proper headers, cookies, and URL parameters.

Below is the code provided by the instructor:

import requests
from pprint import pprint as print
import pandas as pd

url = 'https://datacenter-web.eastmoney.com/api/data/v1/get'
headers = {
    "User-Agent": "your own UA",
    "Referer": "https://data.eastmoney.com/stock/tradedetail/2023-03-03.html",
    "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8"
}
cookies = {
    'Cookie': 'your own Cookie'
}
params = {
    "callback": "",
    "sortColumns": "SECURITY_CODE,TRADE_DATE",
    "sortTypes": "1,-1",
    "pageSize": "50",
    "pageNumber": "1",
    "reportName": "RPT_DAILYBILLBOARD_DETAILSNEW",
    "columns": "SECURITY_CODE,SECUCODE,SECURITY_NAME_ABBR,TRADE_DATE,EXPLAIN,CLOSE_PRICE,CHANGE_RATE,BILLBOARD_NET_AMT,BILLBOARD_BUY_AMT,BILLBOARD_SELL_AMT,BILLBOARD_DEAL_AMT,ACCUM_AMOUNT,DEAL_NET_RATIO,DEAL_AMOUNT_RATIO,TURNOVERRATE,FREE_MARKET_CAP,EXPLANATION,D1_CLOSE_ADJCHRATE,D2_CLOSE_ADJCHRATE,D5_CLOSE_ADJCHRATE,D10_CLOSE_ADJCHRATE,SECURITY_TYPE_CODE",
    "source": "WEB",
    "client": "WEB",
    "filter": "(TRADE_DATE<='2023-03-03')(TRADE_DATE>='2023-03-03')"
}
response = requests.get(url, headers=headers, cookies=cookies, params=params)
data = response.json()['result']['data']
df = pd.DataFrame(data)
print(df)

Running the script produces the expected DataFrame, as shown in the screenshot.

Note that the code generated by ChatGPT may need verification, especially the constructed URL, to ensure it returns the desired data.

Conclusion

This article presented a Python web‑scraping problem, provided a detailed analysis and complete code implementation, and helped the community solve the issue.