Essential Python Libraries for Web Scraping and Data Processing

Network

urllib – network library (stdlib).

requests – network library.

grab – network library (based on pycurl).

pycurl – network library (bindings for libcurl).

urllib3 – Python HTTP library with secure connection pooling, file POST support, high reliability.

httplib2 – network library.

RoboBrowser – simple, Pythonic library for browsing web pages without a separate browser.

MechanicalSoup – Python library for automating interaction with websites.

mechanize – stateful, programmable web browsing library.

socket – low‑level network interface (stdlib).

Unirest for Python – lightweight HTTP library supporting multiple languages.

hyper – HTTP/2 client for Python.

PySocks – actively maintained SocksiPy fork, a direct replacement for the socket module.

Asynchronous Network

treq – requests‑like API built on Twisted.

aiohttp – asyncio‑based HTTP client/server (PEP‑3156).

Web Crawling Frameworks

grab – web crawling framework (based on pycurl/multicur).

scrapy – web crawling framework (based on Twisted), does not support Python 3.

pyspider – powerful crawling system.

cola – distributed crawling framework.

portia – visual crawler built on Scrapy.

restkit – Python HTTP resource toolkit for easy resource access.

demiurge – micro‑framework for crawling based on PyQuery.

HTML/XML Parsers

lxml – high‑performance C‑based HTML/XML library with XPath support.

cssselect – parses DOM trees and CSS selectors.

pyquery – parses DOM trees using jQuery‑style selectors.

BeautifulSoup – pure‑Python HTML/XML parser (less efficient).

html5lib – builds DOM according to WHATWG spec, used by browsers.

feedparser – parses RSS/ATOM feeds.

MarkupSafe – provides safe string escaping for XML/HTML/XHTML.

xmltodict – makes XML feel like JSON in Python.

xhtml2pdf – converts HTML/CSS to PDF.

untangle – simple conversion of XML files to Python objects.

Bleach – sanitizes HTML (requires html5lib).

sanitize – cleans up messy data.

Text Processing

difflib – (standard library) helps with diff comparisons.

Levenshtein – fast Levenshtein distance and string similarity.

fuzzywuzzy – fuzzy string matching.

esmre – regular‑expression accelerator.

ftfy – automatically fixes Unicode text.

unidecode – converts Unicode text to ASCII.

uniout – prints readable characters instead of escaped strings.

chardet – universal character encoding detector for Python 2/3.

xpinyin – converts Chinese characters to pinyin.

pangu.py – adjusts spacing between CJK characters and alphanumerics.

awesome‑slugify – slugify library that preserves Unicode.

python‑slugify – slugify library converting Unicode to ASCII.

unicode‑slugify – generates Unicode slugs.

pytils – simple tools for Russian strings (including transliteration slugify).

PLY – Python implementation of lex and yacc.

pyparsing – generic parser generator framework.

python‑nameparser – parses human name components.

phonenumbers – parses, formats, stores, and validates international phone numbers.

python‑user‑agents – parses browser user‑agent strings.

HTTP Agent Parser – Python HTTP proxy analyzer.

Specific Format File Handling

tablib – exports data to XLS, CSV, JSON, YAML, etc.

textract – extracts text from various file types (Word, PowerPoint, PDF, …).

messytables – parses messy tabular data.

rows – unified data interface supporting many formats (CSV, HTML, XLS, TXT, …).

python‑docx – read, query, and modify Microsoft Word .docx files.

xlwt / xlrd – read/write Excel files and metadata.

XlsxWriter – creates Excel .xlsx files.

xlwings – call Python from Excel and vice‑versa.

openpyxl – read/write Excel 2010 XLSX/XLSM/XLT* files.

Marmir – extracts Python data structures and converts them to spreadsheets.

PDFMiner – extracts information from PDF documents.

PyPDF2 – splits, merges, and transforms PDF pages.

ReportLab – fast creation of rich PDF documents.

pdftables – extracts tables directly from PDFs.

Python‑Markdown – Python implementation of John Gruber’s Markdown.

Mistune – fast, full‑featured pure‑Python Markdown parser.

markdown2 – complete, fast Python Markdown implementation.

PyYAML – YAML parser for Python.

cssutils – CSS library for Python.

feedparser – generic feed parser (also listed under HTML/XML).

sqlparse – non‑validating SQL statement parser.

http‑parser – C‑based HTTP request/response parser.

opengraph – parses Open Graph protocol tags.

pefile – multi‑platform module for parsing Portable Executable files.

psd‑tools – reads Adobe Photoshop PSD files into Python data structures.

Natural Language Processing

NLTK – premier platform for building Python programs that work with human language data.

Pattern – web mining module with NLP tools, machine learning, etc.

TextBlob – consistent API for deeper NLP tasks, built on NLTK and Pattern.

jieba – Chinese word segmentation.

SnowNLP – Chinese text processing library.

loso – another Chinese tokenizer.

genius – conditional random field based Chinese tokenizer.

langid.py – standalone language identification system.

Korean – Korean morphological analysis library.

pymorphy2 – Russian morphological analyzer (POS tagging + inflection).

PyPLN – distributed NLP pipeline written in Python, exposing NLTK via a web API.

Browser Automation & Emulation

selenium – automates real browsers (Chrome, Firefox, Opera, IE).

Ghost.py – wrapper for PyQt’s WebKit (requires PyQt).

Spynner – wrapper for PyQt’s WebKit (requires PyQt).

Splinter – unified API for browser simulation (Selenium driver, Django client, Zope).

Multiprocessing

threading – standard library thread runner, effective for I/O‑bound tasks.

multiprocessing – standard library for running multiple processes.

celery – distributed task queue based on message passing.

concurrent‑futures – high‑level interface for asynchronous execution.

Asynchronous

asyncio – standard library (Python 3.4+) for async I/O, event loops, coroutines, and tasks.

Twisted – event‑driven networking engine.

Tornado – web framework and asynchronous networking library.

pulsar – event‑driven concurrent framework for Python.

diesel – green‑event based I/O framework for Python.

gevent – coroutine‑based network library using greenlet.

eventlet – asynchronous framework with WSGI support.

Tomorrow – syntactic sugar for asynchronous code.

Queues

celery – distributed asynchronous task queue.

huey – lightweight multithreaded task queue.

mrq – Python distributed work queue using Redis & Gevent.

RQ – lightweight Redis‑based task queue manager.

simpleq – simple, infinitely scalable Amazon SQS‑based queue.

python‑gearman – Python API for Gearman.

Cloud Computing

picloud – execute Python code in the cloud.

dominoup.com – cloud execution for R, Python, and MATLAB code.

Email

flanker – email address and MIME parser.

Talon – Mailgun library for extracting quotes and signatures from messages.

URL and Network Address Operations

furl – small library that makes URL manipulation easy.

purl – immutable URL with a clean API for debugging and manipulation.

urllib.parse – parses URLs into components and recombines them.

tldextract – accurately separates TLD and subdomain from a URL.

netaddr – library for displaying and manipulating network addresses.

Web Content Extraction

newspaper – news article extraction and content curation.

html2text – converts HTML to Markdown‑style text.

python‑goose – HTML content/article extractor.

lassie – human‑friendly web content retrieval tool.

micawber – small library for extracting rich content from URLs.

sumy – automatic summarization of text files and HTML pages.

Haul – extensible image crawler.

python‑readability – fast Python interface to arc90 readability tool.

scrapely – library for extracting structured data from HTML pages.

youtube‑dl – command‑line program to download videos from YouTube.

you‑get – YouTube, Youku, Niconico video downloader for Python 3.

WikiTeam – tool for downloading and preserving wikis.

WebSocket

Crossbar – open‑source application messaging router (WebSocket and WAMP implementation in Python).

AutobahnPython – Python implementation of WebSocket and WAMP protocols.

WebSocket‑for‑Python – client and server library for Python 2/3 and PyPy.

DNS Resolution

dnsyo – checks your DNS across more than 1500 global DNS servers.

pycares – interface to c‑ares, a C library for DNS requests and asynchronous name resolution.

Computer Vision

OpenCV – open‑source computer vision library.

SimpleCV – readable interface for cameras, image processing, feature extraction, format conversion (based on OpenCV).

mahotas – fast computer‑image processing algorithms, fully C++ implementation, NumPy‑based arrays.

Proxy Servers

shadowsocks – fast tunnel proxy that helps bypass firewalls (supports TCP/UDP, TFO, multiple users, smooth restart, IP blacklist).

tproxy – simple TCP routing proxy (layer 7) based on Gevent, configured with Python.

Other Python Tools

awesome‑python

pycrumbs

python‑github‑projects

python_reference

pythonidae