Build a Scrapy Spider for dmoz.org in Four Simple Steps

Overview

We use dmoz.org as a target to demonstrate how to build a Scrapy crawler.

Four-Step Process

Create a project : In an empty directory open a command window and run scrapy startproject tutorial. This creates a tutorial folder with the following structure:

scrapy.cfg – project configuration file

tutorial/ – Python module for the project

tutorial/items.py – item definitions

tutorial/pipelines.py – pipeline definitions

tutorial/settings.py – settings

tutorial/spiders/ – directory for spider classes

Define items : Items are containers similar to Python dictionaries, created with scrapy.item.Item and scrapy.item.Field. For this tutorial we define a DmozItem with fields name, url, and description.

Write the spider : A spider inherits from scrapy.spider.BaseSpider and must define three attributes: name (unique identifier), start_urls (list of entry URLs), and parse() (callback to process responses). The example dmoz_spider.py sets allowed_domains, iterates over links, and saves each page to a file. An initial UnicodeDecodeError is resolved by adding a sitecustomize.py file that forces UTF‑8 encoding.

Store results : The simplest storage method uses Scrapy’s feed exports. Running scrapy crawl dmoz -o result.json -t json writes the collected items to a JSON file.

Using XPath Selectors

In the Scrapy shell you can experiment with selectors. Common XPath expressions include: /html/head/title – selects the page title element. //td – selects all <td> elements. //ul[@class="directory-url"]/li – selects list items inside a <ul> with class directory-url, which isolates the desired entries.

Selectors provide methods such as xpath(), css(), extract(), and re() for extracting data.

Putting It All Together

After refining the XPath expression, the spider correctly extracts only the relevant titles without capturing navigation links. The final spider returns DmozItem objects, which are then exported to JSON.

Conclusion

This guide demonstrates a complete workflow for building a Scrapy spider, handling common issues, and extracting structured data from a website.