Build a Python Web Scraper to Grab Maoyan TOP100 Movies in Minutes

For Python beginners, web‑scraping is an ideal first project that provides a sense of achievement. This tutorial walks you through building a basic scraper that fetches the Maoyan TOP 100 movie rankings, covering three core modules: an HTML downloader, an HTML parser, and a data storage component.

1. Construct the HTML Downloader

import requests
from requests.exceptions import RequestException

headers = {'User-Agent': 'Mozilla/5.0 '}

def get_one_page(url):
    try:
        res = requests.get(url, headers=headers)
        if res.status_code == 200:
            return res.text
        return None
    except RequestException:
        return None

2. Construct the HTML Parser

import re

def parse_one_page(html):
    pattern = re.compile(
        r'<dd>.*?board-index.*?>(\d+)</i>.*?data-src="(.*?)".*?name"><a'
        r'.*?>(.*?)</a>.*?star">?(.*?)</p>.*?releasetime">?(.*?)</p>'
        r'.*?integer">?(.*?)</i>.*?fraction">?(.*?)</i>.*?</dd>',
        re.S)
    items = re.findall(pattern, html)
    for item in items:
        yield {
            'index': item[0],
            'image': item[1],
            'title': item[2],
            'actor': item[3].strip()[3:],
            'time': item[4].strip()[5:],
            'score': item[5] + item[6]
        }

3. Construct the Data Storage

import json

def write_to_file(content):
    with open('result.txt', 'a', encoding='utf-8') as f:
        f.write(json.dumps(content, ensure_ascii=False) + '
')
        f.close()

4. Assemble the Main Workflow

from multiprocessing import Pool

def main(offset):
    url = 'http://maoyan.com/board/4?offset=' + str(offset)
    html = get_one_page(url)
    for item in parse_one_page(html):
        print(item)
        write_to_file(item)

if __name__ == '__main__':
    p = Pool()
    p.map(main, [i * 10 for i in range(10)])

Notes:

Use yield instead of return in the parser so the function produces a generator, allowing iteration over all movies without exiting after the first match. re.findall with the re.S flag makes the dot match newlines, enabling multi‑line pattern matching.

Setting ensure_ascii=False in json.dumps preserves Chinese characters in the output file.

Opening the file in append mode ( 'a') prevents overwriting previous results.

Multiprocessing via Pool speeds up crawling by fetching multiple pages concurrently.

Running the script produces a result.txt file containing JSON lines for each movie, and the console prints each parsed item. This simple architecture—downloader, parser, storage—forms the backbone of most web scrapers, whether small or large scale.