Scrape Douban Book Metadata with Python: From Excel Lists to Data Analysis

The article addresses a practical need: given a list of hundreds of book titles in an Excel file, automatically retrieve each book's metadata from Douban (publisher, publication date, ISBN, price, rating, rating count) and integrate it into a pandas DataFrame for analysis.

Page Analysis

Initial attempts to use the standard search URL

https://book.douban.com/subject_search?search_text={title}&cat=1001

failed because the returned HTML contained no data. The author then inspected network traffic and discovered the search suggestion endpoint https://book.douban.com/j/subject_suggest?q={title} returns JSON with title, url, pic, and id fields. The first result is used for simplicity.

JSON Parsing and Detail Page Fetching

For each book title from the Excel 书名 column, the script calls the suggestion API, takes the first result's url, and fetches the book detail page HTML.

XPath Extraction

Using lxml's etree, the author extracts fixed fields via direct XPath:

Book name: //*[@id="wrapper"]/h1/span/text() Rating: //*[@id="interest_sectl"]/div/div[2]/strong/text() Rating count:

//*[@id="interest_sectl"]/div/div[2]/div/div[2]/span/a/span/text()

(with fallback for "评价人数不足")

The info section (publisher, publication date, ISBN, etc.) has variable structure. The author maps its HTML tree: keys appear in <span> nodes, values may be in text nodes or child <a> or <span> nodes, each pair separated by <br>. A custom function getBookInfo(binfo, cc) iterates the parsed nodes, building a dictionary of key-value pairs by detecting span, a, and br tags.

Main Loop and DataFrame Construction

The main loop iterates the title list, calls the suggestion API, fetches the detail page, extracts fields, merges the info dictionary via res.update(getBookInfo(...)), and appends each book's dict to a list rlst. Finally, pd.DataFrame(rlst) creates a DataFrame exported to out_douban_binfo.xlsx.

Data Merging and Statistical Analysis

The scraped DataFrame is left-merged with the original Excel DataFrame bsdf on 书名. Basic stats: 421 books, 309 authors, 97 publishers. Top authors and publishers are listed via value_counts().head(7). Example: querying books by author "吴军" returns four titles with reading dates and publishers.

Time-Series Visualization

Monthly reading counts are derived from 阅读时间 using strftime('%Y-%m'), plotted as a line chart. A pivot table ( pivot_table with np.sum) breaks down monthly counts by year (2016-2018), revealing higher reads in February and July, a pre-July upward trend, and a post-August decline; November 2016 peaked at 40+ books.

Rating Distribution

A boxplot of 评分 shows median ~7.8, 75% of rated books ≥7.2, with some below 4.0. Top-10 rated books are listed (commented code).

Further Analysis Ideas

Word clouds for book titles and authors

Publisher province distribution

Fit word count vs. page count

Normalize multi-currency prices via exchange rates and analyze price distribution

The author notes the code omits extensive validation and error handling, and invites feedback. The approach—analyze problem → build HTML tree → write XPath parsers—is presented as transferable to other scraping tasks.