How to Resolve Common Python Web Scraping Errors and Export Data to CSV

1. Introduction

The author encountered a Python web‑scraping problem in a community group and shares the details to help others. The issue stemmed from missing request headers, causing the crawler to receive no data.

2. Implementation Process

Initially, a simple for loop was used, but the lack of proper request headers prevented data acquisition. After suggestions from experienced members, the author added error handling and refined the code.

The first error handling attempt produced the following traceback (image shown below):

A basic exception handling block was introduced:

res = response.json()
try:
    data = res["data"]
    symbol1 = data["quote"]["symbol"]
    name = data["quote"]["name"]
    current = data["quote"]["current"]
    chg = data["quote"]["chg"]
    percent = data["quote"]["percent"]
    print(symbol1, name, current, chg, percent)
    with open('股票.csv', 'a+', encoding='utf-8') as f:
        f.write('{},{},{},{},{}
'.format(symbol1, name, current, chg, percent))
except:
    print("该股票url无具体信息: ", symbol)

This approach was later refined for better readability and reliability:

if res['data']['tags'] is not None:
    data = res["data"]
    symbol1 = data["quote"]["symbol"]
    name = data["quote"]["name"]
    current = data["quote"]["current"]
    chg = data["quote"]["chg"]
    percent = data["quote"]["percent"]
    print(symbol1, name, current, chg, percent, " ==> 数据下载成功！")
    with open('股票.csv', 'a+', encoding='utf-8') as f:
        f.write('{},{},{},{},{}
'.format(symbol1, name, current, chg, percent))
else:
    print(f"{symbol}无具体信息: ", res)
    
time.sleep(1)

The author notes that the explicit is not None check can be omitted for brevity, and the condition can be simplified to if res['data']['quote']: for clearer output.

3. Conclusion

The article summarizes the Python web‑scraping problem, presents step‑by‑step code solutions, and demonstrates how to handle missing data gracefully while exporting stock information to a CSV file. It also shares common error types to help readers avoid similar pitfalls.