Overview of Popular Python Web Scraping Frameworks

Scrapy is a Python framework designed for extracting structured data from websites, suitable for data mining, information processing, and historical data storage; it can easily crawl product information from sites like Amazon.

PySpider is a powerful Python‑based web crawling system that provides a browser interface for writing scripts, scheduling tasks, viewing results in real time, storing data in common databases, and supports task prioritization and scheduling.

Crawley enables high‑speed crawling of website content, supports relational and non‑relational databases, and can export data in formats such as JSON and XML.

Portia is an open‑source visual crawler that allows users to scrape websites without programming by simply annotating pages, after which it generates spiders to extract data from similar pages.

Newspaper is a library for extracting news articles and performing content analysis; it uses multithreading and supports more than ten languages.

Beautiful Soup is a Python library for parsing HTML or XML documents, offering convenient navigation, searching, and modification of the parse tree, which can save hours of development time.

Grab is a Python framework for building web scrapers, ranging from simple five‑line scripts to complex asynchronous crawlers handling millions of pages, providing an API for HTTP requests and DOM interaction.

Cola is a distributed crawling framework that abstracts away the details of distributed execution; users only need to implement a few functions while tasks are automatically scheduled across multiple machines.

Source: https://blog.51cto.com/13460911/2122398