How to Build a Robust Python Scrapy + Selenium Web Crawler for Forum Data

1. Introduction

Web crawlers (also known as spiders or bots) are programs that automatically fetch web information according to rules. They are essential for big data, finance, machine learning, and many other fields.

2. Project Goal

The objective is to crawl every comment of forum posts into a database, support data updates, prevent duplicate crawling, and handle anti‑scraping measures.

3. Project Preparation

Tools: PyCharm; Libraries: Scrapy, Selenium, pymongo, user_agent, datetime; Target site: http://bbs.foodmate.net; ChromeDriver for Selenium.

4. Project Analysis

4.1 Determine site structure

Identify the loading method (static vs. dynamic) and the hierarchical navigation needed to reach post pages.

4.2 Choose crawling method

Use Scrapy for static pages; combine Selenium when dynamic loading or anti‑scraping is present.

5. Implementation

5.1 Step 1 – Identify site type

The site is a static forum; verification is done by loading the page.

5.2 Step 2 – Determine hierarchy

Three‑level navigation: board → thread list → post page.

5.3 Step 3 – Crawling method

Initially used Scrapy alone, but encountered rate‑limit restrictions; switched to Scrapy + Selenium with headless Chrome to render pages before extraction.

5.4 Step 4 – Data storage format

Define item fields in items.py:

class LunTanItem(scrapy.Item):
    title = Field()
    content_info = Field()
    article_url = Field()
    scrawl_time = Field()
    source = Field()
    type = Field()
    spider_type = Field()

5.5 Step 5 – Database

Store results in MongoDB; use an upsert operation to avoid duplicate entries.

class FMPipeline():
    def __init__(self):
        client = pymongo.MongoClient('localhost')
        db = client.scrapy_FM
        self.collection = db.FM

    def process_item(self, item, spider):
        query = {'article_url': item['article_url']}
        self.collection.update_one(query, {"$set": dict(item)}, upsert=True)
        return item

5.6 Step 6 – Additional settings

Configure request headers, concurrency, pipelines, and other options in settings.py as needed.

6. Result Demonstration

Console output shows crawling progress; the crawler runs with 16 threads for good speed. MongoDB stores each comment with user information.

7. Conclusion

The article demonstrates an end‑to‑end workflow for extracting data from a food forum, covering site analysis, crawling strategy, implementation with Scrapy and Selenium, data storage in MongoDB, and practical tips for handling anti‑scraping and duplicate data.