How to Fix Common Scrapy Installation Errors on Windows

Many users have encountered difficulties installing Scrapy on Windows, so this article consolidates a complete installation tutorial to help avoid confusion.

Scrapy is a popular Python web‑crawling framework. While its basic usage is straightforward, installing it can raise several issues.

1. Standard installation attempt : Open a command prompt and run pip install scrapy. This often fails with errors shown in the screenshots.

2. Missing libxml2/lxml : The failure is usually due to the absence of the libxml2 library. Install the appropriate lxml wheel for your Python version and system architecture from a dedicated Windows‑Python‑package site.

3. Locate the correct lxml wheel : Search for “lxml” on the site, then download the wheel matching your Python version (e.g., lxml‑4.2.1‑cp34‑cp34m‑win_amd64.whl for Python 3.4 64‑bit).

4. Install the lxml wheel : Run pip install lxml-4.2.1-cp34-cp34m-win_amd64.whl in the download directory.

After lxml is installed, re‑run pip install scrapy and the installation should succeed.

5. Verify installation : Use pip list to confirm Scrapy appears in the package list.

Now you can create Scrapy projects and start crawling.

6. Visual C++ Build Tools error : If you see Microsoft Visual C++ 14.0 is required, download and install the Visual C++ Build Tools from Microsoft’s website.

7. Twisted wheel compatibility : Errors like “Twisted‑18.7.0‑cp34‑cp34m‑win_amd64.whl is not a supported wheel on this platform” indicate a mismatch between the wheel and your Python version or architecture. Verify the wheel matches your environment, then rename the .whl file to .zip, extract its contents into your virtual environment’s site‑packages folder, and confirm installation with pip list.

After installing Twisted, you can install Scrapy, Pandas, and other libraries without further issues.

This collection of common Scrapy installation problems and their solutions should help you set up the framework on Windows more smoothly.