Tagged articles
scrapy
91 articles · Page 1 of 1
This tutorial introduces XPath syntax for web scraping, explaining HTML node relationships (parent, child, sibling, ancestor, descendant) and demonstrating key expressions including path operators (/ vs //), attribute selection (@), predicates, wildcards, and union operators with practical examples for element targeting.
XMLXPathdata-extraction
0 likes · 7 min read

This guide walks through resolving the 'No module named win32api' error when running Scrapy on Windows, covering failed pip installs, downloading the correct wheel from UCI's Python libs, a .whl-to-.zip workaround, and the final simple fix using pip install pywin32.
Pythonpypiwin32pywin32
0 likes · 5 min read

This tutorial guides you through creating a Scrapy project on Windows, from activating a virtual environment and running scrapy startproject to examining the generated file structure and understanding the purpose of each core file like settings.py, pipelines.py, and spiders.
PythonTutorialpipelines
0 likes · 6 min read

This tutorial walks through creating a Scrapy spider using the genspider command, verifying the project structure, importing the project into PyCharm, synchronizing the spiders folder, examining the generated spider file, and configuring the Python interpreter to use the project's virtual environment.
PyCharmPythonSpider Tutorial
0 likes · 5 min read

This article explains Scrapy's asynchronous architecture built on Twisted, details its five core components (Engine, Scheduler, Downloader, Spiders, Item Pipeline) and middleware, then walks through creating a project with scrapy startproject, the generated directory structure, and running a spider via scrapy crawl.
Async FrameworkPythonTwisted
0 likes · 7 min read

This article reviews six industrial-grade open-source web scraping tools across three categories—AI-focused extractors (Firecrawl, Crawl4AI), browser automation frameworks (Playwright, Selenium, Crawlee), and the distributed crawler Scrapy—providing a selection guide for building AI knowledge bases, handling dynamic pages, or scaling to millions of pages.
AI data extractionPlaywrightbrowser automation
0 likes · 5 min read

This tutorial walks through installing Scrapy on Windows, resolving frequent errors like missing lxml by using unofficial Windows binaries, fixing Microsoft Visual C++ 14.0 requirement via Build Tools, and a workaround for unsupported Twisted wheels by renaming .whl to .zip and extracting.
PythonTwistedVisual C++
0 likes · 8 min read

This article reviews seven widely used big data collection tools—Flume, Fluentd, Logstash, Chukwa, Scribe, Splunk, and Scrapy—detailing their architectures, supported data sources, extensibility, and typical use cases for efficiently gathering and processing large‑scale data.
ChukwaFluentdFlume
0 likes · 11 min read

This guide introduces seven powerful Python libraries—Pandas, Selenium, Flask, Scrapy, Requests, Faker, and Pillow—explaining their core uses, installation commands, and example code snippets to help developers automate tasks, streamline workflows, and accelerate development.
FakerFlaskPandas
0 likes · 6 min read

Learn how to efficiently scrape web pages using Python by exploring multiple approaches—including simple requests with BeautifulSoup, fast parsing with lxml, dynamic content extraction with Selenium, and large‑scale crawling with Scrapy—complete with installation steps, code snippets, and detailed explanations.
BeautifulSoupPythonrequests
0 likes · 10 min read

This article introduces ten practical Python-based web scraping methods—including requests, Selenium, Scrapy, Crawley, PySpider, aiohttp, asks, vibora, Pyppeteer, and Fiddler‑plus‑Node reverse engineering—explaining their use cases, advantages, and code examples for efficiently gathering e‑commerce and app data.
aiohttpdata collectionscrapy
0 likes · 8 min read

This article outlines Python's advantages for web crawling, introduces core libraries such as Requests, BeautifulSoup, and Scrapy, details a step-by-step development workflow, provides practical code examples for extracting news titles, and highlights important considerations and advanced techniques for robust scraper implementation.
BeautifulSoupPythondata-extraction
0 likes · 5 min read

This article reviews seven widely used Python web frameworks—Django, Flask, Scrapy, Tornado, Web2py, Weppy, and Bottle—detailing their main features, advantages, and drawbacks to help developers choose the most suitable tool for their projects.
DjangoFlaskPython
0 likes · 8 min read

This article compiles twenty‑three Python web‑scraping projects, each described with its purpose, key features, and a direct GitHub repository link, offering developers a ready‑made toolbox for data collection across platforms such as WeChat, DouBan, Zhihu, Bilibili, and more.
GitHubPythondata collection
0 likes · 9 min read

This article introduces ten practical Python techniques—including requests, Selenium, Scrapy, Crawley, PySpider, aiohttp, asks, vibora, Pyppeteer, and Fiddler‑based reverse engineering—to efficiently collect e‑commerce and app data while addressing common challenges such as IP blocking, captchas, and authentication.
aiohttpdata collectionscrapy
0 likes · 8 min read

This article introduces seven practical Python tools—including Pandas, Selenium, Flask, Scrapy, Requests, Faker, and Pillow—explaining their core features, typical use cases, and providing ready‑to‑run code snippets to help developers automate tasks and accelerate project development.
FakerPandaspillow
0 likes · 6 min read

The article examines how Python Requests, Scrapy, and Go’s charset package detect page encodings, reveals why they often mis‑decode Chinese GB‑series pages, and proposes a unified strategy—prefer header charset, then HTML meta, finally a reliable heuristic—to eliminate garbled text in web scraping.
Pythonencodingrequests
0 likes · 8 min read

Scrapy is a powerful Python-based web scraping framework designed for large-scale and complex website data extraction. It offers high-level abstractions, built-in data extraction tools using XPath and CSS selectors, asynchronous processing for parallel requests, and flexible pipelines for data storage, making it ideal for efficient and scalable web scraping projects.
Asynchronous ProcessingPythonbackend development
0 likes · 5 min read

This article explains Scrapy's comprehensive crawling framework and Twisted's event‑driven networking engine, detailing their core concepts, workflow, code execution process, and how to debug Scrapy spiders using breakpoint tracing, providing a deep technical overview for backend developers.
PythonTwistedWeb Crawling
0 likes · 15 min read

This article explains the technical principles and implementation steps of web crawlers, introduces common crawling frameworks, provides a Python example for extracting app store rankings, and then details various anti‑crawling methods such as CSS offset, image camouflage, custom fonts, dynamic rendering, captchas, request signing, and honeypots, followed by counter‑strategies for each.
PythonWeb Crawlinganti‑crawling
0 likes · 24 min read

The article explains web‑crawling basics, Python and Scrapy examples, then surveys common anti‑crawling defenses such as CSS offsets, image camouflage, custom fonts, dynamic rendering, captchas, request signatures and honeypots, and finally presents anti‑anti‑crawling countermeasures—including CSS‑offset reversal, font decoding, headless‑browser rendering and YOLOv5‑based captcha cracking, while stressing legal compliance.
CAPTCHAPythonWeb Crawling
0 likes · 25 min read

This article provides a detailed walkthrough of Scrapy, covering its event‑driven architecture, component interactions, XPath parsing fundamentals, installation steps, project creation, sample spider code, item pipelines, middleware customization, and essential configuration settings for effective web crawling in Python.
PipelineSpiderWeb Crawling
0 likes · 12 min read

This tutorial demonstrates how to build a Scrapy spider that crawls Zhihu user follow and follower data via Zhihu’s public APIs, handles request headers, parses JSON responses, paginates results, and stores the extracted information into MongoDB using a custom item pipeline.
APIMongoDBPython
0 likes · 11 min read

This article introduces the GerapyPlaywright package, explains how to install it, configure Scrapy to use Playwright via middleware and PlaywrightRequest, and provides a complete example spider with code snippets and logging output for JavaScript‑rendered page crawling.
GerapyPlaywrightPlaywrightPython
0 likes · 13 min read

This article explains how font‑based anti‑scraping works, shows how to locate and download custom font files, decode their glyph mappings into a dictionary, and use the mappings to extract real data from a recruitment site with Python, Scrapy and MySQL.
Font Anti‑ScrapingMySQLdata-extraction
0 likes · 12 min read

This article walks you through using Scrapy to fetch price data from Beijing Xinfadi's website, covering request analysis, spider creation, item definition, pagination, data extraction, pipeline setup, and exporting results to CSV with full code examples.
Crawlerbackend developmentdata-extraction
0 likes · 7 min read

This tutorial walks you through the fundamentals of Scrapy, covering its architecture, project setup, spider creation, item and pipeline definitions, pagination techniques, and multiple ways to store scraped data such as JSON files and MongoDB, all with clear code examples.
PipelineSpiderdata-extraction
0 likes · 16 min read

This tutorial walks you through analyzing Tencent's recruitment page, locating the Ajax JSON endpoint, and using Scrapy to create a project, spider, items, pagination, settings, and data export to collect job postings efficiently.
Tencent Jobsdata-extractionscrapy
0 likes · 8 min read

This article provides a comprehensive, step‑by‑step guide to the Scrapy web‑crawling framework, covering its core components, installation methods, project layout, spider creation, data extraction techniques, pagination handling, pipeline configuration, and how to run the crawler to collect and store data.
CrawlerPythondata-extraction
0 likes · 13 min read


21CTOJul 12, 2021 · Backend Development
This comprehensive guide introduces Scrapy's architecture, explains its core components and data flow, teaches XPath fundamentals, walks through installation, project creation, spider coding, item and pipeline definitions, middleware customization, pagination handling, and essential settings for effective Python web crawling.
CrawlerPythonXPath
0 likes · 14 min read

This guide explains how to combine Scrapy and Selenium to scrape dynamically rendered web pages, covering installation, project setup, middleware configuration, Selenium driver handling, and code examples that demonstrate a complete end‑to‑end crawling workflow.
Dynamic PagesPythonmiddleware
0 likes · 12 min read

This article introduces seven open‑source Python web frameworks—Django, Flask, Scrapy, Tornado, Web2py, Weppy, and Bottle—detailing their main features, typical use cases, and the key advantages and disadvantages of each to help developers choose the most suitable framework for their projects.
DjangoPythonWeb Framework
0 likes · 8 min read

This guide demonstrates how to write a Scrapy spider, run it via the command line, use CrawlerProcess and CrawlerRunner for single and multiple spider execution, and explains the observed middleware behavior to help you choose the most reliable method.
CrawlerProcessCrawlerRunnerMultiple Spiders
0 likes · 3 min read

This article introduces seven powerful Python tools—including Pandas, Selenium, Flask, Scrapy, Requests, Faker, and Pillow—explaining their core features and providing ready-to-use code snippets to help developers automate data analysis, testing, web development, crawling, API calls, fake data generation, and image processing.
FakerFlaskPandas
0 likes · 6 min read

This tutorial walks through building a Python web crawler using Scrapy and Selenium to extract forum comments, store them in MongoDB, handle anti‑scraping measures, avoid duplicate data, and demonstrates the full end‑to‑end process with code examples and results.
data-crawlingscrapyweb-scraping
0 likes · 12 min read

This article introduces various web crawling strategies—including focused crawlers, general-purpose crawlers, incremental crawlers, and deep‑web crawlers—explains their underlying principles, presents practical Python code examples for image, e‑commerce and movie data extraction, and discusses deduplication methods and form‑filling techniques.
deep webincremental crawlingscrapy
0 likes · 13 min read

This beginner‑friendly guide walks readers through installing Scrapy, creating a project and spider, running and debugging crawlers, implementing parsing with CSS/XPath, and overcoming common hurdles such as JavaScript rendering, user‑agent spoofing, and proxy rotation via configurable middlewares, enabling quick start of web‑crawling projects.
ProxyPythonWeb Crawling
0 likes · 13 min read

This guide shows how to create a custom Scrapy command to launch every spider at once, separate each spider's settings for better modularity, and resolve video download problems by adjusting request headers and handling file saving correctly.
BackendCustom CommandPython
0 likes · 5 min read

This article provides a step‑by‑step guide on using Scrapy’s crawl template to automatically scrape product details such as ID, title, shop name, shop link, and price from JD.com, including source analysis, project setup, code snippets, and result verification.
JD.comPythonbackend development
0 likes · 4 min read

This tutorial shows how to locate the "next page" link on a website, extract its URL using Scrapy selectors, add proper checks, and integrate the pagination logic into a Scrapy spider so that all article pages are crawled automatically.
CrawlerPythonscrapy
0 likes · 6 min read

This tutorial explains how to use Scrapy's Request objects to feed article detail URLs into the crawler, configure callbacks for parsing, handle relative URLs with urljoin, and yield requests so Scrapy can download pages, completing the core data extraction workflow.
PythonWeb Crawlingrequest
0 likes · 5 min read

This tutorial shows how to configure Scrapy to start from a list page, extract every article link, follow pagination automatically, and parse each article using XPath/CSS selectors, with practical shell commands and visual examples.
CSS selectorsPaginationPython
0 likes · 6 min read

This article demonstrates how to locate and extract cover‑image URLs from a web page using Scrapy, explains handling absolute and relative URLs, shows the necessary XPath and meta‑passing code, and provides debugging tips to verify that the image URL is correctly transferred through the spider.
MetaPythonXPath
0 likes · 6 min read

This tutorial explains how to retrieve an article’s cover image URL by starting from the list page, extracting the first image, and passing its URL through Scrapy’s Request meta dictionary to subsequent parsing callbacks, highlighting why list‑page extraction is more reliable than detail‑page scraping.
MetaPythoncover image
0 likes · 5 min read

This guide walks you through extracting likes, comments, and article content from web pages using Scrapy’s CSS selectors, showing how to locate elements like bookmark buttons, parse numeric data with regular expressions, and integrate the resulting code into your Python spider for reliable data collection.
CSS selectorsPythonregular expressions
0 likes · 7 min read

This article introduces the fundamentals of CSS selectors, compares them with XPath, and demonstrates step‑by‑step how to use CSS selectors in Scrapy to extract titles, dates, tags, likes, and other article data from web pages.
CSS selectorsPythonscrapy
0 likes · 6 min read

This guide walks you through installing Scrapy-Distributed, setting up RabbitMQ and RedisBloom containers, creating a sitemap spider, configuring the distributed scheduler and dupefilter, and running the spider, while explaining why this non‑intrusive solution improves over existing Scrapy‑Redis and scrapy‑rabbitmq approaches.
PythonRabbitMQRedisBloom
0 likes · 7 min read

This article continues a Scrapy tutorial by showing how to extract like counts, comment counts, and full article content using XPath selectors, regular expressions, and debugging techniques, providing step‑by‑step code examples and screenshots to help Python developers automate web data collection.
PythonXPathdata-extraction
0 likes · 6 min read

This tutorial shows how to apply XPath expressions within the Scrapy framework to extract titles, publication dates, tags, content, likes, favorites, and comments from a sample website, providing practical code snippets and tips for reliable web data collection.
PythonWeb CrawlingXPath
0 likes · 5 min read

This article provides a step‑by‑step guide on configuring Scrapy, creating a spider project, extracting comic page URLs and images using BeautifulSoup, handling pagination, and saving the downloaded images locally with Python code.
BeautifulSoupImage Downloadscrapy
0 likes · 14 min read

This article provides a comprehensive step‑by‑step guide to installing Scrapy, understanding its core components and processing flow, creating a weather‑data crawling project, writing items, settings, middlewares, spiders, running the crawler, exporting results, and storing the scraped data into MongoDB.
CrawlerMongoDBPython
0 likes · 15 min read

This tutorial walks through analyzing a proxy‑listing website’s structure, building a Python scraper using requests, Scrapy, regular expressions and BeautifulSoup, extracting IP, port, location and type fields across multiple pages, and saving the collected data to files, illustrating key web‑crawling techniques.
ProxyPythonregular expressions
0 likes · 6 min read

This guide introduces Scrapy, a powerful Python web‑scraping framework, explains its architecture and components, walks through installation, project creation, spider development, query syntax, recursive crawling, and item pipelines, providing practical code examples for building robust crawlers.
CrawlerPythonTwisted
0 likes · 8 min read

This guide outlines responsible web‑crawling practices, covering robots.txt compliance, legal pitfalls such as unauthorized personal data and copyrighted content, recommended request intervals, and relevant Chinese data‑security regulations, helping developers avoid server overloads and potential lawsuits.
Data EthicsWeb Crawlingbackend development
0 likes · 4 min read

This article explains how to build a self‑healing dynamic proxy pool for 24/7 web crawling using Scrapy and Redis, covering requirements, design, implementation details, deployment steps, and a reusable Scrapy middleware example.
proxy poolredisscrapy
0 likes · 7 min read

This guide introduces the Scrapy framework, explains its architecture—including engine, scheduler, downloader, spiders, pipelines, and middlewares—covers installation, project setup, item definition, spider coding, pipeline handling, pagination, and provides practical code examples for extracting data from Douban books.
Item PipelinePythonWeb Crawling
0 likes · 18 min read


21CTOAug 16, 2019 · Backend Development
This guide walks through designing a full‑featured web‑crawler platform, covering rule maintenance, job scheduling, async and real‑time crawling with Scrapy, project setup, item pipelines, settings, local execution, custom parameters, server deployment via Scrapyd, API usage, and fast real‑time crawling with Requests, BeautifulSoup, Flask, and multithreading.
FlaskMultithreadingPython
0 likes · 16 min read

This article explains how to design a flexible web‑crawling platform using Scrapy, covering rule maintenance, job scheduling, asynchronous and real‑time crawlers, project setup, code structure, settings, local execution, deployment with scrapyd, API usage, and examples of Flask‑based real‑time services.
FlaskPythonScrapyd
0 likes · 16 min read

This comprehensive guide explains what web crawlers are, walks through HTTP request/response fundamentals, introduces essential Python modules like requests, re, XPath, BeautifulSoup, and threading, provides practical code examples, and details how to use the Scrapy framework—including its architecture, components, distributed crawling, and useful auxiliary tools.
HTTPPythonscrapy
0 likes · 11 min read

This tutorial walks you through using Scrapy's Xpath selectors to locate and extract titles, dates, comments, and content from web pages, demonstrating both manual and browser‑assisted methods, and shows how to integrate the expressions into your Scrapy spider for reliable data harvesting.
PythonXPathscrapy
0 likes · 6 min read

Learn how to disable robots.txt compliance in Scrapy, use the Scrapy shell for rapid URL debugging, and apply XPath selectors directly in the shell to efficiently extract data, dramatically speeding up development and avoiding repeated full-crawl executions.
PythonXPathdebugging
0 likes · 4 min read

This article introduces practical techniques for debugging Scrapy projects in PyCharm, including creating a main.py launcher, leveraging the built‑in execute function, using breakpoints, and controlling the debug session to streamline development and avoid common path issues.
PyCharmPythonscrapy
0 likes · 5 min read


JavaEdgeMar 21, 2019 · Backend Development
This guide walks through selecting Scrapy over Requests + BeautifulSoup, explains web page types, outlines crawler use‑cases, details regular‑expression syntax and non‑greedy matching, demonstrates practical regex patterns with images, compares depth‑first and breadth‑first crawling, and covers URL deduplication and string‑encoding pitfalls in Python.
PythonWeb Crawlingregex
0 likes · 11 min read

This guide explains why Scrapy on Windows raises the "No module named win32api" error, walks through installing the correct pywin32 package (or pypiwin32), shows how to obtain the proper wheel from an unofficial source, and provides extra tips for locating Scrapy spider names.
Pythonmodule errorpypiwin32
0 likes · 5 min read

This step‑by‑step guide shows how to create a Scrapy spider project for the Jobbole website, configure the project structure, import it into PyCharm, set up the correct Python interpreter, and verify the generated spider code, preparing you for data extraction.
PyCharmPythonSpider
0 likes · 5 min read

This article walks you through setting up a Windows virtual environment, installing Scrapy, creating a new Scrapy project, exploring its directory structure, and opening it in PyCharm, providing clear commands and screenshots for each step.
PythonWeb Crawlingproject setup
0 likes · 6 min read

This article provides a detailed, step‑by‑step tutorial for installing the Python Scrapy framework on Windows, covering virtual environment setup, handling common dependency issues such as Twisted, using alternative mirrors for faster downloads, and verifying the installation, all illustrated with screenshots.
Pythonscrapyvirtualenv
0 likes · 6 min read

This article introduces the Scrapy framework, explains its architecture and five core components, guides you through creating a Scrapy project, configuring spiders, pipelines, and middlewares, and demonstrates how to run the crawler to efficiently collect and process web data using Python.
PythonWeb Crawlingbackend development
0 likes · 7 min read

Learn how to install the Python Scrapy framework on Windows, resolve common libxml2 and Visual C++ dependencies, handle wheel compatibility issues, and verify the installation, with detailed screenshots and step‑by‑step instructions to avoid typical errors.
Pythonscrapy
0 likes · 6 min read

This article walks through building a Python Scrapy spider to extract comprehensive car brand, series, and model data from Autohome, covering environment setup, project initialization, spider and item definitions, handling lazy-loaded pages, CSV output configuration, rate limiting, user‑agent rotation, and debugging tips.
AutohomeCar Datascrapy
0 likes · 10 min read

This guide walks you through step‑by‑step solutions for typical Scrapy installation problems on Windows, covering missing libxml2/lxml wheels, Visual C++ requirements, and Twisted wheel compatibility, so you can get the framework up and running smoothly.
PythonWeb Crawlingscrapy
0 likes · 7 min read

This tutorial walks you through creating a Scrapy project, defining items, writing a spider, and exporting scraped data to JSON while covering common pitfalls like encoding errors and XPath selector usage for extracting titles, URLs, and descriptions from dmoz.org.
Crawlerscrapy
0 likes · 12 min read

This guide introduces Scrapy, a fast Python web‑crawling framework, explains its architecture, installation, project setup, spider creation, execution, and advanced features like XPath selectors, recursion, and item pipelines, providing a complete hands‑on tutorial.
Crawlerbackend developmentscrapy
0 likes · 9 min read

This tutorial explains how to use Scrapy on Ubuntu to create a project, define items, set up pipelines and settings, write a spider, and scrape completed novels from Qidian, while noting the VIP access limitation.
QidianUbuntuWeb Crawling
0 likes · 3 min read


21CTOSep 7, 2018 · Backend Development
This article outlines the major challenges of large‑scale e‑commerce product data extraction—such as ever‑changing site formats, scalable architecture, performance throughput, anti‑bot defenses, and data quality—and shares the hard‑won lessons Scrapinghub gained after crawling over a trillion product pages.
Web Crawlingdata-extractionscale
0 likes · 15 min read

This tutorial shows how to export WeChat Moments using a third‑party service, then build a Python Scrapy spider to crawl the exported pages, parse the JSON data, and save the moments to a file, with detailed commands and code examples.
JSONWeChatdata-extraction
0 likes · 8 min read

This tutorial walks you through obtaining WeChat Moments data via a third‑party export service, setting up a Scrapy project, analyzing the JSON responses, and implementing the spider code to extract and save posts and timestamps.
WeChatdata-extractionscrapy
0 likes · 7 min read

This tutorial walks beginners through setting up a Python Scrapy project, writing a spider to fetch forum thread titles and contents, using XPath for parsing, and enhancing the crawler with pipelines, middleware, and common settings for robust web scraping.
PipelineXPathmiddleware
0 likes · 13 min read

This article walks through the complete Scrapy implementation for extracting WeChat Moments data, covering item definition, spider configuration, request handling, parsing logic, pipeline setup, execution commands, and encoding fixes to produce a clean JSON output.
data-miningscrapyweb-scraping
0 likes · 5 min read

This article explains how to design and build a lightweight Java web crawler framework, covering crawler fundamentals, anti‑scraping challenges, core components such as URL manager, scheduler, downloader, parser and pipeline, and provides concrete code examples and architectural diagrams.
BackendJavaarchitecture
0 likes · 14 min read

This article explains how to create a Python‑based Scrapy spider that logs into Sina Weibo using cookies, crawls user profiles, posts, followers and followees from the WAP site at speeds exceeding 13 million records per day, and stores the data in MongoDB.
MongoDBPythonWeb Crawling
0 likes · 6 min read

This tutorial explains how a Python developer can set up a Scrapy project, write spiders to crawl Zhihu user profiles, store the results in a MySQL database, adjust settings for headers and delays, and finally perform simple gender and location analysis on the collected data.
PythonZhihubackend development
0 likes · 14 min read

Scrapy is a fast, high‑level Python framework for web crawling and data extraction, featuring an asynchronous Twisted engine, modular components like spiders, pipelines, and middlewares, and includes detailed installation steps, project setup, spider creation, query syntax, recursion, and item pipelines for robust scraping.
PythonScrapy Tutorialscrapy
0 likes · 12 min read

Learn how to set up a Scrapy project to crawl comprehensive car brand, series, and model data from Autohome, covering environment preparation, project initialization, spider and pipeline creation, CSV output, rate limiting, and useful debugging tips.
AutohomeCSV exportCar Data
0 likes · 10 min read

This guide walks through building a Python web crawler to extract novel titles and URLs from the qu.la ranking page, explains the site’s clear HTML structure, shows how to deduplicate entries with a set, and provides complete code snippets plus performance tips and a Scrapy upgrade path.
CrawlerPythonlist deduplication
0 likes · 5 min read


ITPUBMay 2, 2017 · Backend Development
This guide explains why websites employ anti‑scraping defenses, outlines the most common header checks such as User‑Agent, Referer, and Cookies, and provides practical Scrapy code snippets for rotating user agents, managing proxies, handling X‑Forwarded‑For, limiting request rates, and dealing with dynamic AJAX content using Selenium or PhantomJS.
HeadersProxyanti-scraping
0 likes · 7 min read

This tutorial walks you through setting up Scrapy on Ubuntu, creating a project, defining items, configuring pipelines and settings, and writing a spider to extract finished novels from the Qidian website, while noting the limitation of VIP‑only content.
PythonQidianUbuntu
0 likes · 3 min read

This tutorial walks you through creating a Scrapy project, defining items, writing a spider, and exporting data to crawl the DMOZ website, covering command‑line setup, XPath extraction, handling encoding errors, and using pipelines for storage.
Item PipelinePythonWeb Crawling
0 likes · 11 min read

This article walks you through the complete installation process for Scrapy, the Python-based web crawling framework, covering prerequisite Python setup, required dependencies like lxml, setuptools, zope.interface, Twisted, pyOpenSSL, win32py, and finally verifying the installation, preparing you for large‑scale data extraction tasks.
PythonWeb Crawlingdata-extraction
0 likes · 4 min read


ITPUBMay 6, 2016 · Backend Development
This guide compares Scrapy (especially version 0.16) with gevent‑based crawling solutions, outlines their strengths, weaknesses, and common pitfalls, and provides practical tips, resource links, and deployment advice for building efficient Python web scrapers.
BackendGeventPython
0 likes · 11 min read

This article compares distributed, Java‑based, and non‑Java web crawlers—examining Nutch, Crawler4j, WebMagic, WebCollector, Scrapy and alternatives—highlighting their strengths, limitations, and suitability for tasks such as data extraction, multi‑threading, AJAX handling, and search‑engine construction.
NutchWeb Crawlingcrawler frameworks
0 likes · 11 min read
