Tagged articles
web crawler
22 articles · Page 1 of 1
This article explains web crawler fundamentals, from the need for automated data collection amid information explosion to the URL-based queue crawling process, basic Python libraries, and the role of frameworks in boosting development efficiency.
PythonURL queuecrawler framework
0 likes · 5 min read

This tutorial walks through creating a Maven‑based Spring Boot backend project with multiple modules, configuring pom.xml files, application properties, and logging, then adds a scheduled Douyin hot‑search crawler using OkHttp, demonstrating full end‑to‑end setup for a web service.
DouyinMavenSpring Boot
0 likes · 31 min read

This article explains Go's native concurrency model, covering lightweight goroutines, channel communication patterns, scheduling details, and demonstrates a practical web‑crawler example that leverages these features for efficient parallel processing.
ConcurrencyGochannel
0 likes · 5 min read

This article explains how to set up a web crawler using PHP and Selenium, covering installation of Selenium and its PHP bindings via Composer, configuring a Chrome WebDriver, simulating user actions to fetch news links, extracting titles and content, and storing results, with tips for further optimization.
PHPautomationselenium
0 likes · 4 min read

This article explains how to create a basic web crawler in a Linux environment using PHP, covering prerequisite installations, script development with cURL and DOMDocument, execution steps, and sample output while emphasizing legal and ethical considerations for web scraping.
DOMDocumentLinuxPHP
0 likes · 4 min read

This guide explains how to create a basic PHP web crawler by using cURL to fetch pages, DOMDocument and XPath to parse HTML, and then storing the extracted data, while also providing a complete example script and reminders about legal and ethical considerations.
Backend DevelopmentDOMDocumentPHP
0 likes · 3 min read

This article describes a PHP‑based web crawler that extracts links and images using regular expressions, stores URLs in MySQL, handles duplicate detection via MD5, discusses performance limitations, and provides the full source code and usage instructions.
Backend DevelopmentMySQLPHP
0 likes · 8 min read

This article walks through a Python web‑crawler problem raised in a community, showing step‑by‑step how to start the project, troubleshoot terminal errors in PyCharm, and verify the directory structure using the tree command, providing a clear solution for beginners.
Pythontroubleshootingweb crawler
0 likes · 3 min read

Photon is a fast, multithreaded Python web crawler that extracts URLs, files, and various intelligence from targets, offering flexible options, Ninja mode, and extensive command‑line parameters while supporting Linux, Windows, macOS, and Termux environments.
Multithreadingcommand-lineinformation security
0 likes · 10 min read

This article demonstrates how to create a Python web crawler that fetches Baidu Baike entries, covering the main program structure, URL manager, page downloader, HTML parser using BeautifulSoup, and output generator, with complete code snippets and sample results.
BeautifulSoupPythonbaidu-baike
0 likes · 8 min read


ITPUBJun 17, 2021 · Information Security
A recent Chinese court case reveals that a university graduate used a custom web‑crawler to harvest more than 1.18 billion Taobao user records, which were then sold to a partner who ran fraudulent WeChat groups, leading to both perpetrators’ conviction for violating personal information protection laws.
Chinadata scrapinginformation security
0 likes · 10 min read

This article details how to build a Node.js web crawler for Indeed job listings, covering entry page selection, HTML parsing with Cheerio, request handling, MongoDB task storage, and a modular architecture that extracts city, category, search, brief, and detail data for a searchable job engine.
BackendMongoDBindeed
0 likes · 15 min read

This article demonstrates a Java‑based web crawler written in Groovy that uses regular‑expression parsing to retrieve paginated company data from a government portal, constructs SQL insert statements, and stores the results in MySQL, with full source code and structural screenshots.
GroovyJavaMySQL
0 likes · 6 min read

This tutorial continues the mica-http complete guide, showcasing the new v1.1.7 release with proxy, retry, page crawling, model visualization, results, documentation links, and open‑source tool recommendations for building efficient backend crawlers.
HTTPJavamica-http
0 likes · 3 min read


21CTOMay 22, 2019 · Fundamentals
This article explains web crawlers—what they are, their classifications, typical use cases, and step‑by‑step workflow—covers the robots protocol, then delves into HTTP and HTTPS fundamentals, request/response structures, common methods, headers, status codes, and the security trade‑offs of HTTPS.
HTTPStatus Codesnetwork protocol
0 likes · 10 min read

This article explains how to design a modular Python web crawler by breaking the system into five core components—scheduler, URL manager, downloader, parser, and data storage—provides detailed code examples for each module, and demonstrates a complete end‑to‑end crawling workflow on a sample website.
Backend DevelopmentPythonScraping
0 likes · 12 min read

This article explains the evolution and classification of web crawlers, describes the design and components of the Mole distributed crawler—including scheduler, fetcher, processor, rate‑limiting, URL deduplication, and Elasticsearch storage optimization—and outlines common anti‑anti‑crawling strategies.
ElasticsearchURL deduplicationanti‑crawling
0 likes · 12 min read

This article examines the evolution of a web crawler architecture across three versions, highlighting the importance of completeness, standardization with UML, clear goals, accuracy, and maintainability to build a scalable and cost‑effective backend system.
BackendSystem DesignUML
0 likes · 6 min read

This guide walks you through creating a lightweight Node.js web crawler using the request and cheerio modules, covering preparation, installation, core code, and testing steps, so you can fetch page HTML, parse data, and store results with just a few dozen lines of code.
Node.jscheeriorequest
0 likes · 5 min read

This article explains how to design and build a lightweight Java web crawler framework, covering crawler fundamentals, anti‑scraping challenges, core components such as URL manager, scheduler, downloader, parser and pipeline, and provides concrete code examples and architectural diagrams.
BackendJavaarchitecture
0 likes · 14 min read


21CTONov 13, 2016 · Backend Development
This guide explains the fundamentals of creating a PHP web crawler, covering server communication basics, interpreting robots.txt and sitemap files, and providing practical code examples using file_get_contents and cURL for efficient content retrieval.
Backend DevelopmentPHPcurl
0 likes · 6 min read


21CTOOct 9, 2015 · Big Data
This article compiles 33 notable open‑source web crawler projects across multiple programming languages, detailing their core features, licensing, supported platforms, and typical use cases, helping developers choose the right tool for large‑scale data harvesting and analysis.
C++C++Java
0 likes · 22 min read
