Master Web Scraping in 12 Lines: Grab Douban Movie Reviews with Python

Introduction

Many students and analysts face difficulties obtaining data and turn to web scraping; this article demonstrates how to explore web scraping with only 12 lines of Python code.

Scraping Target

We use requests + XPath to crawl a portion of short reviews for the movie "Black Panther" on Douban.

Core Code

import requests
from lxml import etree
import pandas as pd
import time
import random
from tqdm import tqdm

name, score, comment = [], [], []

def danye_crawl(page):
    url = 'https://movie.douban.com/subject/6390825/comments?start=%s&limit=20&sort=new_score&status=P&percent_type=' % (page*20)
    response = etree.HTML(requests.get(url).content.decode('utf-8'))
    print('
', '第%s页评论爬取成功' % page) if requests.get(url).status_code == 200 else print('
', '第%s页爬取失败' % page)
    for i in range(1,21):
        name.append(response.xpath('//*[@id="comments"]/div[%s]/div[2]/h3/span[2]/a' % i)[0].text)
        score.append(response.xpath('//*[@id="comments"]/div[%s]/div[2]/h3/span[2]/span[2]' % i)[0].attrib['class'][7])
        comment.append(response.xpath('//*[@id="comments"]/div[%s]/div[2]/p' % i)[0].text)

for i in tqdm(range(11)):
    danye_crawl(i)
    time.sleep(random.uniform(6,9))

res = pd.DataFrame({'name':name, 'score':score, 'comment':comment}, columns=['name','score','comment'])
res.to_csv('豆瓣.csv')

Tool Preparation

Chrome browser for HTTP request analysis and packet capture.

Python 3 with modules: requests (simple HTTP requests), lxml (fast XPath parsing), pandas (data handling), time (set crawl intervals), random (generate random delays), tqdm (progress bar).

Basic Steps

Network request analysis.

Web page content parsing.

Data extraction and storage.

Key Knowledge Points

Crawling protocol (robots.txt, crawl‑delay).

HTTP request analysis.

Using requests for GET requests.

XPath syntax for element selection.

Fundamental Python syntax.

Pandas for data processing.

Crawling Protocol

The robots.txt file defines allowed and disallowed paths; the Crawl-delay directive suggests a polite interval, which we set to a random 6‑9 seconds.

HTTP Request Analysis

Using Chrome DevTools on the Douban short‑review page, we identified the request URL:

https://movie.douban.com/subject/6390825/comments?start=0&limit=20&sort=new_score&status=P&percent_type=

Each subsequent page increments the start parameter by 20. After page 11, login is required, so the demo limits to the first 11 pages.

Requests Usage

We send a GET request, decode the response with UTF‑8, and check the status code (200 indicates success).

XPath Parsing

XPath quickly extracts usernames, scores, and comments from the HTML. Chrome can copy XPath expressions directly.

Data Processing

Extracted data are stored in lists, converted into a dictionary, then a pandas.DataFrame and finally exported to a CSV file.

Conclusion

The requests + XPath approach successfully scrapes Douban short reviews for "Black Panther", providing a solid data foundation for text analysis or other data‑mining tasks. Future articles will cover advanced topics such as custom headers, cookies, login simulation, and distributed crawling.

Plain Code Version

import requests
from lxml import etree
import pandas as pd
import time
import random
from tqdm import tqdm

name, score, comment = [], [], []

def danye_crawl(page):
    url = 'https://movie.douban.com/subject/6390825/comments?start=%s&limit=20&sort=new_score&status=P&percent_type=' % (page*20)
    response = requests.get(url)
    response = etree.HTML(response.content.decode('utf-8'))
    if requests.get(url).status_code == 200:
        print('
', '第%s页评论爬取成功' % page)
    else:
        print('
', '第%s页爬取失败' % page)
    for i in range(1,21):
        name_list = response.xpath('//*[@id="comments"]/div[%s]/div[2]/h3/span[2]/a' % i)
        score_list = response.xpath('//*[@id="comments"]/div[%s]/div[2]/h3/span[2]/span[2]' % i)
        comment_list = response.xpath('//*[@id="comments"]/div[%s]/div[2]/p' % i)
        name.append(name_list[0].text)
        score.append(score_list[0].attrib['class'][7])
        comment.append(comment_list[0].text)

for i in tqdm(range(11)):
    danye_crawl(i)
    time.sleep(random.uniform(6,9))

res = {'name':name, 'score':score, 'comment':comment}
res = pd.DataFrame(res, columns=['name','score','comment'])
res.to_csv('豆瓣.csv')