This tutorial introduces XPath syntax for web scraping, explaining HTML node relationships (parent, child, sibling, ancestor, descendant) and demonstrating key expressions including path operators (/ vs //), attribute selection (@), predicates, wildcards, and union operators with practical examples for element targeting.
This article explains Scrapy's asynchronous architecture built on Twisted, details its five core components (Engine, Scheduler, Downloader, Spiders, Item Pipeline) and middleware, then walks through creating a project with scrapy startproject, the generated directory structure, and running a spider via scrapy crawl.
The article explains the fundamental difference between parsing—converting documents into AI‑friendly formats that preserve structure and context—and extraction—pulling predefined fields into structured outputs—while offering concrete scenarios, decision criteria, and example implementations with LlamaParse and LlamaExtract.
Faced with 30,000 report images containing 300,000 rows of tabular data, the author explains why WPS failed at scale, analyzes the OCR error sources, and shares a Python script that batch‑crops images to boost recognition accuracy before exporting everything to Excel.
The author details how they used AI‑assisted Python scripts to batch‑process over 30,000 report images, extract tables via OCR, crop noisy regions, merge results into a single Excel sheet with 99% accuracy, and automate validation, eliminating manual data entry.
This guide explains how OpenClaw’s built‑in browser feature and the optional Browser Relay extension enable effortless data extraction from both public and login‑required websites, providing step‑by‑step installation commands, Chrome extension loading instructions, and a practical example of scraping top posts from a social platform.
This tutorial shows how to build a JD.com search URL, fetch the page with urllib, parse the HTML using BeautifulSoup, and reliably extract each product's name, link, image and price while handling missing image URLs and avoiding regex complexity.
This guide explains PHP’s array_column() function—available since version 5.5—detailing its syntax, parameters, return value, and practical code examples that show how to extract specific keys from multidimensional arrays and optionally use another key as the resulting array’s index.
This article introduces the Scrapeasy Python library, shows how to install it, initialize website and page objects, retrieve links, images, videos, and other file types, and download them with simple one‑line commands, providing a fast way to extract data from any site.
This tutorial demonstrates how to use Python and Selenium to collect thousands of hot comments from QQ Music, parse the data from dynamically loaded list items, store it in CSV files, and finally generate a word‑cloud visualization of the comment content.
This article introduces DrissionPage, a Python library that merges Selenium and Requests, explains its three page objects, highlights seamless mode switching, built‑in utilities, API changes in version 4.0, and provides practical code examples for web automation, data crawling, and testing.
This article introduces Scrapeasy, a Python library that lets you scrape websites, retrieve subpage links, images, PDFs, and videos with just a few lines of code, showing installation, basic usage, and advanced download techniques for various file types.
This article walks through a Python web‑scraping issue, shows how to clean scraped data using regular expressions, provides sample code for extracting both floating‑point and integer numbers from mixed strings, and offers a concise solution that readers can apply to their own projects.
This tutorial explains why converting legacy .doc files to .docx is necessary, shows how to install LibreOffice and python-docx, provides a Python script to perform the conversion via LibreOffice's command‑line interface, and demonstrates reading and modifying tables in the resulting .docx files.
This article walks through a Python web‑scraping problem, explains why the original script produced incorrect results, presents a corrected implementation with full code examples, and clarifies the underlying bug caused by mutable data reuse.
This article walks you through using Playwright in Python to scrape the Xinfadi market website, showing how to capture URLs, handle request payloads, extract JSON data, and process results with a complete, runnable code example.
This tutorial walks you through building a Python web‑scraper for Beike (Lianjia) second‑hand property listings, covering session spoofing, dynamic parameters, pagination, multithreaded detail fetching, data cleaning, and exporting results to Excel in a step‑by‑step manner.
This article explains why proxy IPs are essential for reliable web crawling, compares dynamic and static residential proxies, and provides step‑by‑step Python code to scrape product titles, prices and links from Snapdeal while demonstrating how to integrate proxies for improved efficiency and security.
This article outlines Python's advantages for web crawling, introduces core libraries such as Requests, BeautifulSoup, and Scrapy, details a step-by-step development workflow, provides practical code examples for extracting news titles, and highlights important considerations and advanced techniques for robust scraper implementation.
This article explains the PHP array_column() function introduced in version 5.5, detailing its syntax, parameters, return values, and provides multiple code examples showing how to extract specific keys from multidimensional arrays, including using an index key to create associative arrays.
This tutorial demonstrates how to use Apache Tika in a Java project to parse PDF, Word, and text documents, extract specific fields such as name and ID number, and shows the required Maven dependencies and sample code for performing the extraction.
This article introduces JSONPath as a query language for JSON, explains its core concepts and basic syntax with examples, demonstrates various practical use cases such as data extraction, API testing, and shows tool support across Python, Java, and JavaScript.
This guide shows how to use JSONPath within a Spring Boot 3.4.0 project to efficiently query, filter, and modify complex JSON structures, providing Maven setup, syntax overview, operator tables, and practical code examples for extracting authors, prices, books, and applying custom filters.
This tutorial explains e‑commerce web scraping fundamentals, covering definitions, tool types, data categories, step‑by‑step Python script creation with Requests, BeautifulSoup, and Selenium, provides sample code for Amazon, Walmart, and eBay, discusses challenges like dynamic pages and anti‑scraping measures, and recommends using specialized scraping APIs.
This article walks through a real‑world Python question about extracting numeric work‑experience values from a CSV column, presenting four distinct regex‑based solutions with complete code snippets, visual results, and a concise summary for data‑analysis practitioners.
This article compares Python and PHP for web scraping, outlining each language's strengths, ecosystem, performance, learning curve, and community support to help readers decide which tool best fits their project requirements and experience level.
This article walks you through using Python's requests and lxml libraries to crawl a fund website, extract stock IDs, fetch detailed fund information, process the data, and save it to CSV for further analysis, complete with code snippets and screenshots.
This article demonstrates how to create a reusable PHP function that searches a two‑dimensional array for a specific id and returns the value of a given key, such as the title, using a simple loop and conditional check.
This tutorial shows how to use the jq command‑line tool to extract each user's name from a JSON array, illustrating simple commands such as jq '.users[].name' data.json, describing the author's move from manual IDE methods, and emphasizing the importance of knowing jq exists and leveraging AI to generate scripts from plain‑language descriptions.
Learn how to quickly scrape Amazon product details such as images, ASIN, name, price, rating, and image URLs by leveraging Google Sheets' ImportFromWeb function, requiring only a few clicks in a spreadsheet without writing any code, and see step‑by‑step instructions with screenshots.
This article walks through a Python web‑scraping project that fetches skin images, names, and prices from Honor of Kings, covering project background, required tools, site analysis, code implementation, data processing, and legal considerations.
This article explains what .nc (netCDF) files are, why they are useful for storing scientific time‑series data, and provides two clear Python solutions—standard reading and direct extraction—complete with installation tips, code examples, and practical guidance.
This article walks through a real‑world Python data‑processing problem where a messy Excel column is parsed into separate numeric fields, presenting two Pandas solutions—one using regular expressions and another using string splitting—along with complete code examples and practical tips.
This article demonstrates how regular expressions can be applied in API automation testing through ten practical Python examples that extract URLs, validate emails, parse JSON, match phone numbers, detect sensitive words, retrieve HTTP status codes, verify dates, parse query parameters, enforce password strength, and extract XML tag content.
This article explains a frequent Pandas warning, demonstrates why the code still runs, and provides step‑by‑step solutions with screenshots to help Python users fix the issue and ensure correct data extraction.
This article presents a collection of useful Python scripts—including file sorting, empty‑folder removal, batch renaming, web‑scraping, bulk image downloading, form automation, and various text‑processing utilities—each accompanied by clear explanations and ready‑to‑run code examples.
This tutorial demonstrates how to use the tika-python library to extract textual content from PDF and Excel files, providing code examples and important notes about installation and potential formatting limitations, and suggestions for further processing to obtain readable or structured output.
This tutorial explains how to create a simple Flask web site and RESTful API, then demonstrates retrieving the table data using Excel's web import feature and Python's requests, BeautifulSoup, and pandas libraries, while comparing the two approaches.
This guide introduces awk as a powerful text‑processing tool, covering its basic syntax, field separators, pattern matching, arithmetic operations, built‑in variables, custom functions, control flow, and how to process both files and command output with clear code examples.
This article introduces Scrapeasy, a Python library that enables one‑line web scraping, media extraction, and file downloading, and provides step‑by‑step code examples for installing the package, initializing websites, retrieving links, images, videos, and PDFs, making data collection fast and easy.
This article walks through a Python web‑scraping tutorial that extracts ETF codes and names from Eastmoney, detailing the problem, step‑by‑step code using requests, pandas, and handling pagination, and concludes with a brief promotion of a data‑collection service.
This article walks through retrieving ETF fund codes and names from Eastmoney using Python's requests and pandas, explains constructing the correct URLs, handling pagination, cleaning the JSON response, and provides complete sample scripts, while also highlighting a simpler solution and recommending a data‑collection platform.
This article explains how to write a Python crawler that uses the WeChat public platform's open API to obtain an access token, retrieve the list of articles from a specified public account, and extract each article's title, summary, and link for further analysis.
This article explains the JSON Pointer syntax, provides encoding rules and practical examples, demonstrates how to extract values from complex JSON structures using Jackson's JsonUtil in Java, and shows how to map nodes to Java beans, offering a concise guide for developers.
This article compiles a variety of Python web‑scraping utilities—including file download assistants, video downloaders, proxy IP pool builders, captcha bypass scripts, and game data extractors—providing descriptions, installation commands, usage examples, download links, and reference links for each tool.
This article walks through a Python automation solution for extracting invoice data from PDFs, presenting the code implementation, expected results, and practical tips for handling similar data‑processing questions in community groups.
This guide explains how to quickly capture network traffic from a mobile app using the Stream proxy, export the data as a HAR file, and then decode and parse the compressed responses with a short Python script to extract the desired JSON information.
Scrapy is a powerful Python-based web scraping framework designed for large-scale and complex website data extraction. It offers high-level abstractions, built-in data extraction tools using XPath and CSS selectors, asynchronous processing for parallel requests, and flexible pipelines for data storage, making it ideal for efficient and scalable web scraping projects.
This article walks through a real Python web‑scraping problem, explains why missing request headers cause failures, and provides complete, runnable code snippets—including header setup, cookie handling, and request loops—to retrieve stock data from Xueqiu, followed by a concise summary of the solution.
This article traces the development of web crawlers from their 1990s origins to modern implementations, examines their multifaceted value in search, data analysis, and archiving, outlines technical, ethical, and legal challenges for both crawler creators and target sites, and presents practical strategies to mitigate malicious crawling.
This article provides a thorough introduction to the AWK language, covering its basic syntax, common usage patterns, operators, built‑in variables, and numerous practical examples for processing and extracting data from text files.
This tutorial explains how to collect 2022 county‑level GDP and public budget data from the Chinese National Bureau of Statistics using Java's Jsoup library, transform the HTML tables into structured Excel files with EasyExcel, and provides complete source code and step‑by‑step analysis.
This article walks through a Python-based web scraper that extracts job listings from 51job.com, detailing required header and cookie settings, pagination logic, data parsing, and CSV output, and includes full code snippets and tips for handling site changes and further data analysis.
This article walks through a real‑world Python web‑scraping case, showing how to retrieve disease‑pest data from the Shandong government site using custom headers, cookies, and request parameters, and provides two complete code examples that successfully return the desired JSON results.
This article walks through building a Python web scraper that fetches stock trading data using the requests library and pandas, showing the complete code, how to set headers and cookies, and the resulting DataFrame, while highlighting the limits of relying solely on ChatGPT‑generated snippets.
This article walks through a Python web‑scraping solution that fetches GDP data from a website using the requests library, parses HTML with lxml, and demonstrates two approaches—manual XPath extraction and a streamlined pandas.read_html method—while providing complete code snippets and tips for handling pagination and data storage.
This article presents thirteen advanced Python scripts covering tasks such as internet speed testing, Google searching, web automation, lyric retrieval, EXIF extraction, OCR, image cartoonization, recycle bin cleaning, image enhancement, Windows version detection, PDF-to-image conversion, hex-to-RGB conversion, and website status checking, each with code examples.
This article demonstrates how to extract the 'name' field from a Pandas DataFrame column using regular expressions, presenting three code solutions and explaining their usage to help readers solve similar data‑parsing challenges.
This article introduces the Scrapeasy Python library, showing how a single line of code can scrape websites, retrieve links, images, PDFs, and videos, and download them to local storage, making web data extraction fast and straightforward.
This article demonstrates how to read a CSV file with Pandas, use regular expressions to extract tag names from a string column, and transform the results into separate rows using the explode function, while also presenting alternative community‑sourced approaches.
This article walks through several Python techniques—including list splitting, combined list‑processing methods, and regular‑expression extraction—to efficiently parse mixed text and URL strings, illustrating each approach with code snippets and results.
This tutorial explains the fundamentals of web crawling by distinguishing static and dynamic pages, outlining a four‑step process for scraping static sites, and providing a complete Python example that extracts historical weather data, parses HTML with BeautifulSoup, and stores results in CSV files.
This article explains what a web crawler is, describes the basic environment and tools needed for Python crawling, outlines the typical scraping workflow, and presents three implementation styles—basic, function‑encapsulated, and concurrent—illustrated with diagrams and practical guidance.
This article introduces JsonPath as a DSL for reading JSON data in Java, demonstrates Maven integration, provides code examples for extracting and renaming JSON fields, and mentions the jq command‑line tool for additional JSON manipulation.
This tutorial demonstrates how to use regular expressions and pandas string methods to extract numeric values from Series columns, calculate average years of experience, and convert salary strings into numeric monthly figures for further analysis.
This article demonstrates how to retrieve specific values from a pandas DataFrame with a MultiIndex using both straightforward reset‑index methods and native multi‑level indexing techniques such as chained .loc calls, tuple indexing, and level swapping.
This article walks through a Python‑based solution for parsing a list of Chinese address strings, extracting province abbreviations, grouping records by province, and optionally using regular expressions or pandas for further analysis, complete with full source code examples.
This article introduces the renowned "awesome" GitHub repository, highlights its extensive sub‑lists for various domains, focuses on the awesome‑web‑scraping collection, and presents a newly created Chinese version that aggregates Python, JavaScript, Go, and other language‑specific web‑scraping tools and libraries.
This article explains how to scrape data protected by SVG background‑image anti‑scraping by using Selenium to retrieve the SVG URL, parsing the SVG with requests‑html to map background offsets to characters, replacing SVG nodes with text, and finally extracting structured information such as phone numbers and reviews.
This tutorial walks through using Selenium to log into Alibaba International, scrape supplier phone numbers and related details across multiple pages, save the data to CSV, download product images, and finally embed those images into an Excel workbook for easy reference.
This article introduces Jsoup, an open‑source Java library for extracting and manipulating HTML, explains its key features such as DOM traversal and CSS selectors, and provides a concise code example that fetches Wikipedia headlines, helping developers automate web data collection.
This article demonstrates how to use Python's Selenium WebDriver to log into Alibaba International, scrape company contact information across multiple pages, save the data to CSV, download product images, and embed them into an Excel file, providing a complete automation workflow.
This article explains how font‑based anti‑scraping works, shows how to locate and download custom font files, decode their glyph mappings into a dictionary, and use the mappings to extract real data from a recruitment site with Python, Scrapy and MySQL.
This article introduces Python 3 methods for retrieving various document types—including TXT, CSV, PDF, and Word files—using urllib, regular expressions, and file‑specific processing steps, providing practical code examples and workflow guidance for building effective web crawlers.
The article explains Baidu Wenku's document conversion pipeline, detailing how various office formats are transformed into PDF layout data, then into adaptive flow data for mobile devices, and describes the technical methods for extracting structured content and chart metadata from PDFs and OOXML documents.
This article walks you through using Scrapy to fetch price data from Beijing Xinfadi's website, covering request analysis, spider creation, item definition, pagination, data extraction, pipeline setup, and exporting results to CSV with full code examples.
This tutorial walks you through the fundamentals of Scrapy, covering its architecture, project setup, spider creation, item and pipeline definitions, pagination techniques, and multiple ways to store scraped data such as JSON files and MongoDB, all with clear code examples.
This tutorial walks you through analyzing Tencent's recruitment page, locating the Ajax JSON endpoint, and using Scrapy to create a project, spider, items, pagination, settings, and data export to collect job postings efficiently.
This tutorial shows how to capture the network packets of the Super Course Table Android app, log in by sending the required headers and encrypted credentials, and continuously fetch user‑generated topics using Python's urllib2 and JSON parsing, enabling infinite scrolling data extraction.
This tutorial explains how to install and use the Python jsonpath library to efficiently query and extract values from deeply nested JSON structures, covering basic syntax, indexing, conditional filtering, and result formatting with clear code examples and illustrations.
This article provides a comprehensive, step‑by‑step guide to the Scrapy web‑crawling framework, covering its core components, installation methods, project layout, spider creation, data extraction techniques, pagination handling, pipeline configuration, and how to run the crawler to collect and store data.
This tutorial walks you through analyzing a JavaScript‑driven website, discovering the hidden API, configuring request headers with Postman, and writing a Python scraper that extracts book titles, authors, codes, and prices from epubit.com.
This tutorial walks you through installing Selenium and ChromeDriver, configuring the environment, and using Python code to automatically navigate Dangdang's bestseller pages, extract book details with pyquery, and save the results into a CSV file for further analysis.
This tutorial walks beginners through the complete process of building a Python web scraper to collect internship listings from Shixi.com, covering page analysis, static vs dynamic detection, data location with XPath, pagination handling, and storing results in a CSV file using pandas.
Learn step-by-step how to use Python's Selenium library to automate login, scroll through QQ Space, extract historical posts and download album photos, including driver setup, code snippets for login, shuoshuo retrieval, and photo downloading, enabling you to preserve nostalgic content.
This article presents a practical collection of Python regular‑expression techniques for extracting HTML elements such as table rows, links, titles, images, and scripts, showing how to filter tags and handle URL parameters during web crawling.
This guide demonstrates how to use Python's Selenium library to log into a GPS web system, locate elements via name and XPath, handle iframes, click list items, and extract dynamic data, including detailed installation steps for Selenium and ChromeDriver.
This article explains why Python has become a favorite language for developers, introduces the fundamentals of web crawlers, details how they work using Python libraries, and highlights practical uses and advantages, especially when running on Huawei Cloud services.
This article walks through building a Python web crawler that extracts forum post comments into MongoDB, covering project goals, environment setup, site structure analysis, Scrapy and Selenium integration, data storage design, handling anti‑scraping measures, and performance optimization with multithreading.
This tutorial explains how to analyze the Ant Short‑Term Rental website's anti‑scraping mechanisms, extract the required Cookie and User‑Agent headers, and use Python's urllib2 and BeautifulSoup to reliably crawl rental listings, save the data to CSV, and optionally extend the scraper with Selenium.
This article walks you through building a Python web‑scraper that queries Baidu for the keyword “grain,” extracts article titles, abstracts and links using Requests and lxml, saves the results to a text file, and shows the execution output with screenshots.
This beginner‑friendly guide walks readers through installing Scrapy, creating a project and spider, running and debugging crawlers, implementing parsing with CSS/XPath, and overcoming common hurdles such as JavaScript rendering, user‑agent spoofing, and proxy rotation via configurable middlewares, enabling quick start of web‑crawling projects.
This tutorial walks you through installing Scrapy on Windows, creating a project and spider, configuring settings, and using XPath to crawl and extract joke titles and contents from the Qiushibaike website, providing a solid foundation for Python web scraping.
This tutorial explains how to use Python's requests, lxml, and urllib libraries to search Baidu Tieba by keyword, bypass anti‑crawling measures, extract image and video URLs with XPath, and save the media files locally, complete with code examples and screenshots.
This article provides a step‑by‑step guide on using Scrapy’s crawl template to automatically scrape product details such as ID, title, shop name, shop link, and price from JD.com, including source analysis, project setup, code snippets, and result verification.
This article continues a Scrapy tutorial by showing how to extract like counts, comment counts, and full article content using XPath selectors, regular expressions, and debugging techniques, providing step‑by‑step code examples and screenshots to help Python developers automate web data collection.
This tutorial shows how to apply XPath expressions within the Scrapy framework to extract titles, publication dates, tags, content, likes, favorites, and comments from a sample website, providing practical code snippets and tips for reliable web data collection.
This article walks through creating a Python web scraper that extracts financial data for over three thousand listed companies, starting from a simple pandas script and progressively adding error handling, MySQL storage, and multiprocessing to build a robust, production‑ready tool.
This tutorial demonstrates how to use JsonPath in Java to retrieve specific items and slices from JSON arrays, covering zero‑based indexing, negative indices for reverse lookup, Python‑style slice syntax, and provides complete code examples with console output.
Learn how to build a Python web scraper that fetches upcoming movie details from Maoyan.com, covering environment setup, URL pagination, random User‑Agent handling, HTML parsing with XPath, data extraction, and result display, while highlighting best practices and limitations.