This guide demonstrates how to programmatically pull date, title, and document number fields from thousands of similarly structured Word tables, normalize the dates, and export the data into a formatted Excel spreadsheet using Python's docx, datetime, and openpyxl libraries.
Learn how to install the Web Scraper Chrome extension, configure sitemaps and selectors, and handle pagination, scrolling, and click‑load scenarios to build robust, multi‑level web crawling projects without writing code, covering selector types, multi‑page navigation, and data export to CSV.
This article provides a comprehensive introduction to Python web scraping, covering the fundamental concepts of spiders, HTTP/HTTPS protocols, the Requests library usage, custom headers, proxies, cookies, and various data extraction techniques such as JSON parsing, XPath, and regular expressions.
This tutorial shows how to extract a target computer's Chrome browsing history with Python, save it as a text file, and automatically send it via email using a QQ SMTP account, detailing the required environment, scripts, and common pitfalls.
This guide introduces Scrapy, a powerful Python web‑scraping framework, explains its architecture and components, walks through installation, project creation, spider development, query syntax, recursive crawling, and item pipelines, providing practical code examples for building robust crawlers.
This guide introduces the Scrapy framework, explains its architecture—including engine, scheduler, downloader, spiders, pipelines, and middlewares—covers installation, project setup, item definition, spider coding, pipeline handling, pagination, and provides practical code examples for extracting data from Douban books.
This guide walks you through the fundamentals of building a web crawler in Python, covering how to fetch pages with the Requests library, extract data using regular expressions and XPath, and provides practical code examples for each step.
The article shares the author's journey from manual data collection to mastering Java web crawlers, explains why Java is preferred over Python, outlines the five-step crawling workflow, covers essential Java basics, HTTP fundamentals, and provides code examples for URL queuing, time parsing, and timestamp conversion.
An open‑source Java crawler that fetches historical‑today events from a public API is presented, detailing three practical challenges—GET request length limits, ambiguous JSON value types, and month string construction—along with a full code example and a GitHub repository link for reference.
This article demonstrates a Java‑based web crawler written in Groovy that uses regular‑expression parsing to retrieve paginated company data from a government portal, constructs SQL insert statements, and stores the results in MySQL, with full source code and structural screenshots.
This guide explains core XPath concepts—including node selection shortcuts, predicate filters, wildcard usage, multiple path unions, axis navigation, and built‑in functions—providing clear examples for effective XML/HTML data extraction.
This article explains how to update and use a Java HttpClient‑based spider that removes duplicate links, handles legacy page formats, extracts movie metadata and download URLs (magnet, ed2k, Baidu Pan), and stores the results in a MySQL database, with complete source code examples.
The article walks through a Java solution for extracting multilingual data from an HTML table using regular expressions, handling spacing and encoding issues, splitting fields, and constructing INSERT statements to populate a country_code database table.
This tutorial walks you through logging into Taobao with Python requests, handling anti‑scraping measures, extracting product information via the PC search API, parsing JSON data, and saving the results to Excel, while also covering common pitfalls like sliders and proxy management.
This tutorial introduces BeautifulSoup, a powerful Python library for parsing HTML and XML, covering installation, basic usage, tag selection, attribute extraction, navigation of parent and sibling nodes, method and CSS selectors, and best‑practice recommendations for efficient web data extraction.
Learn how to install and use Beautiful Soup 4 in Python to parse HTML, navigate the document tree, access tags, attributes, and text, and perform powerful searches with methods like find_all, CSS selectors, and traversal techniques for effective web scraping.
This guide explains XPath fundamentals, covering node selection syntax, predicates for filtering, wildcard usage, combining multiple paths, axis navigation, and built‑in functions, providing clear examples for each to help developers efficiently query XML or HTML documents.
This article introduces Beautiful Soup, a Python library for parsing HTML/XML into a navigable tree, covering installation, object initialization, tag navigation, attribute handling, searching techniques like find_all, CSS selectors, and practical code examples for effective web data extraction.
As online information explodes, traditional data collection methods fall short, prompting the rise of Python web crawlers that use URLs and libraries like urllib, urllib2, and re, while frameworks boost efficiency, enabling fast, accurate, and automated extraction of web data for analysis.
This guide explains the core XPath syntax—including node selectors, predicates, wildcards, multiple paths, axes, and built‑in functions—providing clear examples so you can efficiently locate and filter XML/HTML elements when scraping web pages.
This guide explains how to extract novel titles and links from a structured ranking website, remove duplicate entries using a set, handle HTML tags, and improve crawling speed with multithreading or the Scrapy framework, all while keeping the code modular and reusable.
This article walks through building a high‑performance novel‑site crawler using MongoDB, detailing the extraction of category links, multi‑process/thread URL management, database operations, and the final deduplication results, while illustrating each step with diagrams and screenshots.
This article explains why regular expressions are essential for Python web crawling, introduces the special characters ^, ., and *, and demonstrates their use with clear code examples and output screenshots, helping readers quickly grasp regex fundamentals for extracting patterns from HTML content.
This article outlines the major challenges of large‑scale e‑commerce product data extraction—such as ever‑changing site formats, scalable architecture, performance throughput, anti‑bot defenses, and data quality—and shares the hard‑won lessons Scrapinghub gained after crawling over a trillion product pages.
This tutorial shows how to export WeChat Moments using a third‑party service, then build a Python Scrapy spider to crawl the exported pages, parse the JSON data, and save the moments to a file, with detailed commands and code examples.
This article introduces Beautiful Soup, a Python library for parsing HTML/XML into a navigable tree, covering installation, object initialization, tag and attribute access, tree traversal, searching techniques like find_all, find, CSS selectors, and practical code examples.
This tutorial walks you through obtaining WeChat Moments data via a third‑party export service, setting up a Scrapy project, analyzing the JSON responses, and implementing the spider code to extract and save posts and timestamps.
This guide demonstrates how to programmatically log into the Super Schedule mobile app using Python, capture its JSON responses, and continuously fetch user-generated topics by mimicking the app’s HTTP requests, complete with sample code and request headers.
This comprehensive guide explains Awk’s role as a versatile, stream‑oriented editor and pattern‑matching language, covering its command‑line syntax, records and fields, scripts, patterns, regular expressions, operators, statements, built‑in variables, functions, I/O handling, and practical code examples for Linux users.
This article introduces AWK—a versatile text‑analysis language—explaining its origins, core concepts, command‑line usage, built‑in variables, printing functions, programming constructs, conditionals, loops, and associative arrays, and provides practical Linux examples for extracting and summarizing data.
This tutorial shows how to use Python's urllib and XPath to accurately extract product details such as name, link, image, and price from JD.com search results, offering a clearer alternative to regular expressions and BeautifulSoup.
This article explains how web crawlers work—from initial URL seeding and request handling to flow control, content extraction, and handling dynamic pages—while covering essential modules, HTTP details, common obstacles like JavaScript rendering, anti‑scraping measures, and strategies for large‑scale, distributed crawling.
This article walks you through using Python's Beautiful Soup library to extract chapter links and text from an online novel, covering installation, basic parsing methods, single‑chapter scraping, and a complete‑book crawler that saves the content to a local file.
This guide walks you through analyzing the Meituan Waimai app's comment section, uncovering the AJAX requests that load data, constructing the proper URL pattern, looping through pages to fetch JSON, and storing the results in files or a database for further analysis.
This guide walks through the complete process of crawling the Sichuan University Public Administration College website to extract every news article, covering target identification, rule definition, code implementation, handling pagination, and troubleshooting missing items.
This tutorial explains how to use Python to log into Sina Weibo, download your favorite posts, comments, images, and video links, save the raw HTML, and outlines the steps and script needed to automate the extraction and later processing of the collected data.
This tutorial walks you through creating a Scrapy project, defining items, writing a spider, and exporting data to crawl the DMOZ website, covering command‑line setup, XPath extraction, handling encoding errors, and using pipelines for storage.
This article walks you through the complete installation process for Scrapy, the Python-based web crawling framework, covering prerequisite Python setup, required dependencies like lxml, setuptools, zope.interface, Twisted, pyOpenSSL, win32py, and finally verifying the installation, preparing you for large‑scale data extraction tasks.
This article explores the fundamentals and advanced techniques of building web crawlers, covering crawler types, essential features, RSS/ATOM harvesting, custom scraping methods, PHP header manipulation, regex extraction, and concurrency, providing actionable code examples for backend developers.
This article explains what web crawlers are, why they’re essential for content recommendation systems, the technical approaches across languages, practical use‑cases like price monitoring and news aggregation, and best practices for building efficient, ethical crawlers.
This article explains how to design and implement a distributed web‑crawling framework in Java that can collect, structure, and store massive amounts of data while handling anti‑scraping measures, duplicate detection, and real‑time monitoring.
This article compiles 33 notable open‑source web crawler projects across multiple programming languages, detailing their core features, licensing, supported platforms, and typical use cases, helping developers choose the right tool for large‑scale data harvesting and analysis.
This article provides a concise, one‑minute introduction to AWK, covering its origin, line‑by‑line processing principle, basic syntax of pattern‑action, built‑in variables and functions, operators, control structures, and how to interact with the shell, illustrated with practical command‑line examples.
