How to Fix Common Regex Errors in Python Web Scraping: A Step-by-Step Guide

1. Introduction

The author, a Python enthusiast, received a question about a Python web‑scraping script that uses requests and regular expressions to parse Douban movie listings.

2. Implementation Process

Original code (shown below) contained a malformed regular‑expression pattern, causing extraction failures.

import requests
import re
url = "https://movie.douban.com/top250"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
}
resp = requests.get(url, headers=headers)
resp.encoding = "utf-8"
pageSource = resp.text
print(pageSource)                 # re.S can make . match newline
obj = re.compile(r'<div class="item">.*?<span class="title">(?P<name>.*?)</sp'
                r'an>.*? <p class=""></p>.*?导演:(?P<dao>.*?) <br>'
                r'(?P<year>.*?) ', re.S)
result = obj.finditer(pageSource)
for item in result:
    print(item.group("name"))
    print(item.group("dao"))
    print(item.group("year"))

After reviewing the pattern, the instructor corrected the regular expression, aligning the tags and removing stray characters. The updated script now successfully prints each movie’s name, director, and release year.

3. Summary

The article demonstrates how to troubleshoot and fix a Python web‑scraping script that relies on regular expressions, providing the corrected code and explaining the underlying issues. It also thanks the community members who contributed to the discussion.