Essential Python Libraries for Web Scraping and Data Processing

This list compiles Python libraries for web crawling and data processing.

Network

urllib – standard library network module.

requests – popular HTTP library.

grab – network library based on pycurl.

pycurl – libcurl bindings.

urllib3 – HTTP library with connection pooling and file upload support.

httplib2 – network library.

RoboBrowser – browser‑like library without a real browser.

MechanicalSoup – library for automating interaction with websites.

mechanize – stateful programmable web‑browser library.

socket – low‑level network interface (stdlib).

Unirest for Python – lightweight HTTP library.

hyper – HTTP/2 client.

PySocks – maintained SocksiPy replacement for the socket module.

Async

treq – requests‑like API built on Twisted.

aiohttp – asyncio‑based HTTP client/server (PEP‑3156).

Web Crawler Frameworks

grab – crawler framework based on pycurl/multicur.

scrapy – crawler framework based on Twisted (no Python 3 support).

pyspider – powerful crawling system.

cola – distributed crawling framework.

Other

portia – visual crawler built on Scrapy.

restkit – HTTP resource toolkit.

demiurge – micro‑framework based on PyQuery.

HTML/XML Parsers

lxml – high‑performance C‑based HTML/XML library with XPath support.

cssselect – CSS selector parser for DOM trees.

pyquery – jQuery‑style selector for DOM trees.

BeautifulSoup – pure‑Python HTML/XML parser (less efficient).

html5lib – WHATWG‑compliant HTML/XML parser.

feedparser – RSS/ATOM feed parser.

MarkupSafe – safe string handling for XML/HTML/XHTML.

xmltodict – treats XML like JSON.

xhtml2pdf – converts HTML/CSS to PDF.

untangle – simple XML‑to‑object conversion.

Cleaning

Bleach – HTML sanitiser (requires html5lib).

sanitize – cleans messy data.

Text Processing

difflib – standard library for diff comparisons.

Levenshtein – fast Levenshtein distance and similarity.

fuzzywuzzy – fuzzy string matching.

esmre – regex accelerator.

ftfy – fixes Unicode text.

Conversion

unidecode – Unicode to ASCII transliteration.

Character Encoding

uniout – prints readable characters instead of escaped strings.

chardet – universal character encoding detector.

xpinyin – converts Chinese characters to pinyin.

pangu.py – adjusts spacing between CJK and alphanumerics.

Slugification

awesome-slugify – Unicode‑preserving slug generator.

python-slugify – Unicode to ASCII slug generator.

unicode-slugify – creates Unicode slugs.

pytils – simple Russian string utilities (including slugify).

General Parsers

PLY – Python implementation of lex and yacc.

pyparsing – generic parsing framework.

Human Names

python-nameparser – parses personal name components.

Phone Numbers

phonenumbers – parses, formats, stores and validates international numbers.

User‑Agent Strings

python-user-agents – parses browser user‑agent strings.

HTTP Agent Parser – parses HTTP proxy strings.

Specific Format File Handling

tablib – export data to XLS, CSV, JSON, YAML, etc.

textract – extracts text from various file types (Word, PPT, PDF, …).

messytables – parses messy tabular data.

rows – unified data interface supporting many formats (CSV, HTML, XLS, TXT, …).

Office

python-docx – read, query, and modify .docx files.

xlwt / xlrd – read/write Excel files and format information.

XlsxWriter – create .xlsx files.

xlwings – call Python from Excel and vice‑versa.

openpyxl – read/write Excel 2010+ files.

Marmir – extract Python data structures to spreadsheets.

PDF

PDFMiner – extract information from PDF documents.

PyPDF2 – split, merge, and transform PDF pages.

ReportLab – generate rich PDF documents.

pdftables – extract tables directly from PDFs.

Markdown

Python-Markdown – implementation of John Gruber’s Markdown.

Mistune – fast, full‑featured pure‑Python Markdown parser.

markdown2 – fast Markdown implementation.

YAML

PyYAML – YAML parser.

CSS

cssutils – CSS library.

ATOM/RSS

feedparser – generic feed parser.

SQL

sqlparse – non‑validating SQL statement parser.

HTTP

http-parser – C‑based HTTP request/response parser.

Micro‑formats

opengraph – parses Open Graph protocol tags.

Portable Executables

pefile – parses and works with PE files on multiple platforms.

PSD

psd-tools – reads Adobe Photoshop PSD files into Python structures.

Natural Language Processing

NLTK – comprehensive platform for processing human language data.

Pattern – web mining module with NLP tools and machine learning.

TextBlob – consistent API built on NLTK and Pattern.

jieba – Chinese word segmentation.

SnowNLP – Chinese text processing.

loso – another Chinese segmentation library.

genius – CRF‑based Chinese segmentation.

langid.py – standalone language identification.

Korean – Korean morphological analysis library.

pymorphy2 – Russian morphological analyzer.

PyPLN – distributed NLP pipeline built with Python.

Browser Automation and Emulation

selenium – automates real browsers (Chrome, Firefox, Opera, IE).

Ghost.py – wrapper for PyQt WebKit (requires PyQt).

Spynner – wrapper for PyQt WebKit (requires PyQt).

Splinter – generic API for browser simulation (supports Selenium, Django client, Zope).

Multiprocessing

threading – standard library threads (good for I/O‑bound tasks).

multiprocessing – standard library for multi‑process execution.

celery – distributed asynchronous task queue.

concurrent‑futures – high‑level interface for asynchronous execution.

Asynchronous

asyncio – standard library for async I/O, event loops, coroutines (Python 3.4+).

Twisted – event‑driven networking engine.

Tornado – web framework and async networking library.

pulsar – event‑driven concurrent framework.

diesel – green‑event based I/O framework.

gevent – coroutine‑based network library using greenlet.

eventlet – async framework with WSGI support.

Tomorrow – syntactic sugar for async code.

Queues

celery – distributed async task queue.

huey – lightweight multithreaded task queue.

mrq – Redis & Gevent based distributed work queue.

RQ – lightweight Redis‑based queue manager.

simpleq – simple, infinitely scalable Amazon SQS‑based queue.

python‑gearman – Gearman API.

Cloud Computing

picloud – execute Python code in the cloud.

dominoup.com – cloud execution for R, Python, and MATLAB.

Email

flanker – email address and MIME parser.

Talon – Mailgun library for extracting quotes and signatures.

URL and Network Address Operations

URL

furl – small library for easy URL manipulation.

purl – immutable URL with a clean debugging API.

urllib.parse – split and combine URL components.

tldextract – accurately separates TLD and subdomains.

Network Address

netaddr – display and manipulate network addresses.

Web Content Extraction

HTML page text and metadata

newspaper – news and article extraction.

html2text – converts HTML to Markdown.

python-goose – HTML/article extractor.

lassie – human‑friendly web content retrieval.

micawber – extracts rich content from URLs.

sumy – automatic summarization of text and HTML pages.

Haul – extensible image crawler.

python‑readability – fast interface to arc90 readability.

scrapely – extracts structured data from HTML pages.

Video

youtube‑dl – command‑line tool to download YouTube videos.

you‑get – YouTube, Youku, Niconico video downloader for Python 3.

Wiki

WikiTeam – tools to download and preserve wikis.

WebSocket

Crossbar – open‑source application messaging router (Python implementation of Autobahn WebSocket and WAMP).

AutobahnPython – WebSocket and WAMP protocol implementation.

WebSocket‑for‑Python – client and server library for Python 2/3 and PyPy.

DNS Resolution

dnsyo – checks DNS across more than 1500 global servers.

pycares – interface to c‑ares for asynchronous DNS queries.

Computer Vision

OpenCV – open‑source computer‑vision library.

SimpleCV – readable interface for camera, image processing, feature extraction (built on OpenCV).

mahotas – fast C++‑based image‑processing algorithms using NumPy arrays.

Proxy Servers

shadowsocks – fast tunnel proxy supporting TCP/UDP, TFO, multiple users, and IP blacklists.

tproxy – simple TCP routing proxy (layer 7) based on Gevent.

Other Python Tools

awesome-python

pycrumbs

python‑github‑projects

python_reference

pythonidae