Explore the Ultimate Python Library Collection for Web Crawling and Data Processing

Network-related

urllib - network library (standard library)

requests - network library

grab - network library (based on pycurl)

pycurl - network library (binds libcurl)

urllib3 - thread‑safe connection pool, file post support, high‑availability HTTP library

httplib2 - network library

RoboBrowser - simple, pythonic library for accessing web pages without a separate browser

MechanicalSoup - library for automated website interaction

mechanize - stateful, programmable web browsing library

socket - low‑level network interface (standard library)

Unirest for Python - lightweight multi‑language HTTP library

hyper - Python HTTP/2 client

PySocks - maintained fork of SocksiPy, can replace the socket module

Asynchronous

treq - API similar to requests based on Twisted

aiohttp - asyncio HTTP client/server (PEP‑3156)

Web Crawlers

grab - web crawling framework (based on pycurl/multicurl)

scrapy - web crawling framework (based on Twisted)

pyspider - powerful crawling system

cola - distributed crawling framework

HTML/XML Parsing

lxml - efficient HTML/XML processing library with XPath support, written in C

cssselect - parses DOM tree and CSS selectors

pyquery - parses DOM tree and jQuery selectors

BeautifulSoup - low‑performance HTML/XML processing library

html5lib - WHATWG‑compliant HTML/XML parser

feedparser - parses RSS/ATOM feeds

MarkupSafe - safe string escaping for XML/HTML/XHTML

xmltodict - treats XML like JSON

Text Processing

difflib - difference calculation tool (standard library)

Levenshtein - fast edit‑distance and string‑similarity calculator

fuzzywuzzy - fuzzy string matching

esmre - regex accelerator

ftfy - automatically fixes Unicode text

Conversion

unidecode - converts Unicode text to ASCII

Slugification

awesome-slugify - Unicode‑preserving Python slugify library

python-slugify - converts Unicode to ASCII slug

unicode-slugify - slug generation tool

pytils - small utilities for Russian strings (includes slugify)

General Parsers

PLY - Python lex and yacc parsing tools

pyparsing - generic framework for building parsers

Office

python-docx - read, query and modify Microsoft Word docx files

xlwt / xlrd - read and write Excel files

XlsxWriter - create Excel .xlsx files

xlwings - BSD‑licensed library for Excel‑Python interaction

openpyxl - read/write Excel 2010 xlsx/xlsm/xltx/xltm files

Marmir - extracts Python data structures and converts them to tables

PDF

PDFMiner - extracts information from PDF documents

PyPDF2 - split, merge, convert PDF files

ReportLab - quickly creates large PDF documents

pdftables - accurately extracts tables from PDF files

Markdown

Python-Markdown - implementation of John Gruber's Markdown

Mistune - fast, full‑featured pure‑Python Markdown parser

markdown2 - fast Markdown implementation in pure Python

YAML

PyYAML - YAML parser for Python

CSS

cssutils - CSS library for Python

ATOM/RSS

feedparser - generic feed parser

SQL

sqlparse - non‑validating SQL statement parser

HTTP

http-parser - C implementation of HTTP request/response parser

Microformats

opengraph - parses Open Graph protocol tags

Portable Executables

pefile - multi‑platform module for parsing PE files

PSD

psd-tools - reads Adobe Photoshop PSD files into Python data structures

Natural Language Processing

NLTK - leading Python NLP library

Pattern - web mining module with NLP tools and machine learning

TextBlob - API for deeper NLP tasks, built on NLTK

jieba - Chinese word segmentation

SnowNLP - Chinese text processing library

loso - Chinese segmentation library

genius - conditional random field based Chinese segmentation

langid.py - independent language identification system

Korean - Korean morphological library

pymorphy2 - Russian morphological analyzer

PyPLN - distributed NLP pipeline built on NLTK

langdetect - Google language detection port

Browser Automation

selenium - automates real browsers (Chrome, Firefox, Opera, IE)

Ghost.py - QtWebKit wrapper (requires PyQT)

Spynner - programmatic web browsing with AJAX support

Splinter - generic API browser simulator (selenium, Django client, Zope)

Headless Tools

xvfbwrapper - Python wrapper to run display in X virtual framebuffer (Xvfb)

Multiprocessing

threading - Python standard library for multithreading (effective for I/O‑bound tasks)

multiprocessing - standard library for multiple processes

celery - distributed message‑driven asynchronous task queue

concurrent-futures - high‑level interface for asynchronous execution of callables

Async

asyncio - asynchronous I/O, event loop, coroutines and tasks (standard library from Python 3.4)

Twisted - event‑driven networking engine

Tornado - web framework and async network library

pulsar - event‑driven concurrent framework

diesel - Greenlet‑based I/O framework

gevent - coroutine‑based Python networking library

eventlet - WSGI‑compatible async framework

Tomorrow - magic for async code

Queues

celery - distributed asynchronous task/ job queue

huey - small multithreaded task queue

mrq - Redis & Gevent based distributed work queue

RQ - lightweight Redis‑based task queue manager

simpleq - simple, infinitely scalable queue based on Amazon SQS

python‑gearman - Python API for Gearman

Cloud Computing

picloud - execute Python code in the cloud

dominoup.com - execute R, Python and MATLAB code in the cloud

Email

flanker - email and MIME handling library

Talon - Mailgun library for extracting quotes and signatures

URL and Network Address

furl - small library for simplifying URL manipulation

purl - simple immutable URL and clean API for debugging

urllib.parse - splits and joins URL components, resolves relative URLs

tldextract - accurately separates registered domain and subdomain using public suffix list

Web Content Extraction

newspaper - news, article extraction and content curation

html2text - converts HTML to Markdown‑style text

python‑goose - HTML content/article extractor

lassie - human‑friendly web content retrieval tool

micawber - small library to extract rich content from URLs

sumy - automatic summarization of text files and HTML pages

Haul - extensible image crawler

python‑readability - fast Python interface to arc90 readability tool

scrapely - library for extracting structured data from HTML pages

libextract - extracts data from websites

Video

youtube‑dl - small command‑line tool to download videos from YouTube

you‑get - Python 3 video downloader for YouTube, Youku, Niconico, etc.

Wiki

WikiTeam - tool to download and preserve wikis

WebSocket

Crossbar - open‑source application message router (Python, Autobahn, WebSocket, WAMP)

AutobahnPython - Python implementation of WebSocket and WAMP protocols

WebSocket‑for‑Python - WebSocket client and server library for Python 2, 3 and PyPy

DNS

dnsyo - checks your DNS on over 1500 global DNS servers

pycares - interface to c‑ares for asynchronous DNS queries

Computer Vision

OpenCV - open‑source computer vision library

SimpleCV - concise, readable interface for camera, image processing, feature extraction (based on OpenCV)

mahotas - fast image processing algorithms implemented in C++ and NumPy arrays

Proxy Servers

shadowsocks - fast tunnel proxy to bypass firewalls (supports TCP, UDP, TFO, multi‑user, smooth restart, IP blacklist)

tproxy - simple TCP routing proxy (layer 7) based on Gevent, configurable in Python

Miscellaneous

awesome‑python

pycrumbs

python‑github‑projects

python_reference

pythonidae