How to Scrape GDP Data with Python and Save to CSV – Full Code Walkthrough

1. Introduction

The author received a request to modify a Python web‑scraping script that originally used pandas to fetch GDP data and write it to a CSV file. The existing code was shared via a link.

2. Implementation

The revised script imports requests, lxml.etree, csv, and pandas, defines gdpData(page) to request each pagination page, and processes the HTML with XPath to extract ranking, region, GDP, and year. The data is written to data.csv using csv.DictWriter. The script also determines the total number of pages and iterates over them.

import requests
from lxml import etree
import csv
import time
import pandas as pd

def gdpData(page):
    url = f'https://www.hongheiku.com/category/gdjsgdp/page/{page}'
    headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'}
    resp = requests.get(url, headers=headers)
    data(resp.text)

def data(text):
    file = open('data.csv', mode='a', encoding='utf-8', newline='')
    csv_write = csv.DictWriter(file, fieldnames=['排名', '地区', 'GDP', '年份'])
    csv_write.writeheader()
    e = etree.HTML(text)
    lst = e.xpath('//*[@id="tablepress-48"]/tbody/tr[@class="even"]')
    for l in lst:
        no = ''.join(l.xpath('./td[1]/center/span/text()'))
        name = ''.join(l.xpath('./td[2]/a/center/text()')[0])
        team = ''.join(l.xpath('./td[3]/center/text()'))
        year = ''.join(l.xpath('./td[4]/center/text()'))
        data_dict = {'排名': no, '地区': name, 'GDP': team, '年份': year}
        print(f'排名：{no} 地区:{name} GDP:{team} 年份:{year} ')
        csv_write.writerow(data_dict)
    file.close()

url = 'https://www.hongheiku.com/category/gdjsgdp'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'}
resp = requests.get(url, headers=headers)
data(resp.text)

e = etree.HTML(resp.text)
count = e.xpath('//div[@class="pagination pagination-multi"][last()]/ul/li[last()]/span/text()')[0].split(' ')[1]
for index in range(int(count) - 1):
    gdpData(index + 2)

3. Conclusion

The provided script successfully extracts the required GDP information and saves it into a CSV file, offering a clear example of Python web‑scraping and data storage for similar tasks.