Python Basics, Common Pitfalls, and a Simple Web Scraper for Douban Book Ratings

The guide starts with a concise overview of Python's conceptual hierarchy, explaining how expressions create objects, statements contain expressions, logical units form functions or classes, modules are .py files, packages are directories with an __init__.py, and programs consist of multiple packages and files.

It then lists ten typical beginner errors, such as poor variable naming, confusion between numeric and string operations, differences between lists and dictionaries, indexing issues, misuse of range(), mixing assignment and comparison operators, infinite loops, and misunderstandings of function return versus print statements.

Following the pitfalls, the article demonstrates how to create a simple web scraper that fetches book ratings from Douban. First, it imports the required modules:

import requests
from bs4 import BeautifulSoup
print('成功导入模块')

It then shows code snippets for extracting HTML tags, printing titles and links, and iterating over URLs to collect data:

# 提取标签
# print(soup.head)  # 头部信息
print(soup.title)  # 标题
print(soup.a)      # 提取的第一个a标签

The scraper defines a function get_data(ui) that sends a GET request, parses the page with BeautifulSoup, locates the book details, and assembles a list of dictionaries containing the book name, rating, additional info, and summary:

def get_data(ui):
    ri = requests.get(url=ui)
    soupi = BeautifulSoup(ri.text, 'lxml')
    infors = soupi.find_all('div', class_="detail-frame")
    lst = []
    for i in infors:
        dic = {}
        dic['书名'] = i.find('h2').text.replace('
', '')
        dic['评分'] = i.find_all('p')[0].text.replace('
', '').replace(' ', '')
        dic['其他信息'] = i.find_all('p')[1].text.replace('
', '').replace(' ', '')
        dic['简介'] = i.find_all('p')[2].text.replace('
', '').replace(' ', '')
        lst.append(dic)
    return lst

url = 'https://book.douban.com/latest'
result = get_data(url)
print(result[:3])

The extracted data is then converted into a pandas DataFrame for further analysis:

import pandas as pd
df = pd.DataFrame(result)
print(df)

Sample output shows a list of dictionaries with book titles, scores, publication details, and brief introductions, illustrating how the scraper gathers and structures real-world data.