This tutorial walks through creating a Maven‑based Spring Boot backend project with multiple modules, configuring pom.xml files, application properties, and logging, then adds a scheduled Douyin hot‑search crawler using OkHttp, demonstrating full end‑to‑end setup for a web service.
This article explains Go's native concurrency model, covering lightweight goroutines, channel communication patterns, scheduling details, and demonstrates a practical web‑crawler example that leverages these features for efficient parallel processing.
This article explains how to set up a web crawler using PHP and Selenium, covering installation of Selenium and its PHP bindings via Composer, configuring a Chrome WebDriver, simulating user actions to fetch news links, extracting titles and content, and storing results, with tips for further optimization.
This article explains how to create a basic web crawler in a Linux environment using PHP, covering prerequisite installations, script development with cURL and DOMDocument, execution steps, and sample output while emphasizing legal and ethical considerations for web scraping.
This guide explains how to create a basic PHP web crawler by using cURL to fetch pages, DOMDocument and XPath to parse HTML, and then storing the extracted data, while also providing a complete example script and reminders about legal and ethical considerations.
This article describes a PHP‑based web crawler that extracts links and images using regular expressions, stores URLs in MySQL, handles duplicate detection via MD5, discusses performance limitations, and provides the full source code and usage instructions.
This article walks through a Python web‑crawler problem raised in a community, showing step‑by‑step how to start the project, troubleshoot terminal errors in PyCharm, and verify the directory structure using the tree command, providing a clear solution for beginners.
Photon is a fast, multithreaded Python web crawler that extracts URLs, files, and various intelligence from targets, offering flexible options, Ninja mode, and extensive command‑line parameters while supporting Linux, Windows, macOS, and Termux environments.
This article demonstrates how to create a Python web crawler that fetches Baidu Baike entries, covering the main program structure, URL manager, page downloader, HTML parser using BeautifulSoup, and output generator, with complete code snippets and sample results.
A recent Chinese court case reveals that a university graduate used a custom web‑crawler to harvest more than 1.18 billion Taobao user records, which were then sold to a partner who ran fraudulent WeChat groups, leading to both perpetrators’ conviction for violating personal information protection laws.
This article details how to build a Node.js web crawler for Indeed job listings, covering entry page selection, HTML parsing with Cheerio, request handling, MongoDB task storage, and a modular architecture that extracts city, category, search, brief, and detail data for a searchable job engine.
This article demonstrates a Java‑based web crawler written in Groovy that uses regular‑expression parsing to retrieve paginated company data from a government portal, constructs SQL insert statements, and stores the results in MySQL, with full source code and structural screenshots.
This tutorial continues the mica-http complete guide, showcasing the new v1.1.7 release with proxy, retry, page crawling, model visualization, results, documentation links, and open‑source tool recommendations for building efficient backend crawlers.
This article explains web crawlers—what they are, their classifications, typical use cases, and step‑by‑step workflow—covers the robots protocol, then delves into HTTP and HTTPS fundamentals, request/response structures, common methods, headers, status codes, and the security trade‑offs of HTTPS.
This article explains how to design a modular Python web crawler by breaking the system into five core components—scheduler, URL manager, downloader, parser, and data storage—provides detailed code examples for each module, and demonstrates a complete end‑to‑end crawling workflow on a sample website.
This article explains the evolution and classification of web crawlers, describes the design and components of the Mole distributed crawler—including scheduler, fetcher, processor, rate‑limiting, URL deduplication, and Elasticsearch storage optimization—and outlines common anti‑anti‑crawling strategies.
This article examines the evolution of a web crawler architecture across three versions, highlighting the importance of completeness, standardization with UML, clear goals, accuracy, and maintainability to build a scalable and cost‑effective backend system.
This guide walks you through creating a lightweight Node.js web crawler using the request and cheerio modules, covering preparation, installation, core code, and testing steps, so you can fetch page HTML, parse data, and store results with just a few dozen lines of code.
This article explains how to design and build a lightweight Java web crawler framework, covering crawler fundamentals, anti‑scraping challenges, core components such as URL manager, scheduler, downloader, parser and pipeline, and provides concrete code examples and architectural diagrams.
This guide explains the fundamentals of creating a PHP web crawler, covering server communication basics, interpreting robots.txt and sitemap files, and providing practical code examples using file_get_contents and cURL for efficient content retrieval.
This article compiles 33 notable open‑source web crawler projects across multiple programming languages, detailing their core features, licensing, supported platforms, and typical use cases, helping developers choose the right tool for large‑scale data harvesting and analysis.
