Fix Common Scrapy Installation Errors on Windows: lxml, Visual C++, Twisted Workarounds

Installing Scrapy on Windows: Common Issues and Fixes

Scrapy is a popular Python web crawling framework. While installing via pip install scrapy often fails on Windows due to missing dependencies.

Issue 1: Missing lxml Dependency

Running pip install scrapy may fail with an error indicating missing libxml2. The solution is to install the lxml binary from the unofficial Windows Python package repository (https://www.lfd.uci.edu/~gohlke/pythonlibs/).

Steps:

Visit the repository and search for "lxml".

Download the wheel matching your Python version and architecture (e.g., lxml-4.2.1-cp34-cp34m-win_amd64.whl for Python 3.4 64-bit).

Install it via pip install lxml-4.2.1-cp34-cp34m-win_amd64.whl.

Then run pip install scrapy again.

Issue 2: Microsoft Visual C++ 14.0 Required

Another common error:

distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 is required

. This occurs because the system lacks the Visual C++ Build Tools.

Solution: Download and install the Visual C++ Build Tools from http://landinghub.visualstudio.com/visual-cpp-build-tools. The package is large but can be removed after Scrapy installs successfully.

Issue 3: Twisted Wheel Not Supported on This Platform

Error:

Twisted-18.7.0-cp34-cp34m-win_amd64.whl is not a supported wheel on this platform

. Even if the wheel matches your Python version and architecture, pip may reject it.

Workaround: Rename the .whl file to .zip, extract its contents into the virtual environment's site-packages directory, then verify with pip list.

Place the downloaded wheel in the virtual environment's site-packages folder.

Rename Twisted-18.7.0-cp34-cp34m-win_amd64.whl to Twisted-18.7.0-cp34-cp34m-win_amd64.zip.

Extract all files to the same folder.

Run pip list to confirm Twisted appears.

After resolving these issues, Scrapy and other packages like Pandas install smoothly. The article notes that environments vary, so other problems may arise.