Master Scrapy: Build a Complete DMOZ Crawler in Four Simple Steps

We use dmoz.org as the target site for a Scrapy crawling tutorial.

Four main steps: create a project, define items, write a spider, and store results via pipelines.

1. Create Project

In an empty directory open a command window and run scrapy startproject tutorial. This creates a tutorial folder with files such as scrapy.cfg, tutorial/items.py, tutorial/pipelines.py, tutorial/settings.py, and tutorial/spiders/.

2. Define Items

Items act like Python dictionaries. Define a class DmozItem with fields name, url, and description in items.py.

3. Write Spider

Implement a spider subclassing scrapy.Spider with required attributes name, start_urls, and a parse() method. The spider fetches pages, extracts data using XPath selectors, and yields DmozItem objects.

Example command to run the spider: scrapy crawl dmoz. If an UnicodeDecodeError occurs, add a sitecustomize.py in the Python site-packages directory to set the default encoding.

4. Store Data

Export results with scrapy crawl dmoz -o dmoz.json -t json or use a custom Item Pipeline for more complex processing.

The tutorial also demonstrates using the Scrapy shell to test XPath expressions, such as //ul[@class="directory-url"]/li, and shows how to extract titles, descriptions, and links from the DMOZ pages.