Tagged articles
XPath
70 articles · Page 1 of 1
This analysis explores Scrapling, a Python scraping library whose adaptive element matching automatically relocates target elements after website redesigns, combining five fetcher tiers, anti-bot evasion, Scrapy-like crawling, and AI agent integration to turn site redesigns from disasters into minor maintenance.
CSS selectorsCloudflareMCP Server
0 likes · 29 min read

A step-by-step Python tutorial demonstrating how to scrape book details (publisher, ISBN, ratings) from Douban using search suggestion API and XPath parsing, then merge with Excel data for statistical analysis and visualization with pandas and matplotlib.
DoubanJupyter NotebookPandas
0 likes · 14 min read

This tutorial introduces XPath syntax for web scraping, explaining HTML node relationships (parent, child, sibling, ancestor, descendant) and demonstrating key expressions including path operators (/ vs //), attribute selection (@), predicates, wildcards, and union operators with practical examples for element targeting.
XMLXPathdata-extraction
0 likes · 7 min read

This article reviews the four main Python selectors used for web scraping—regular expressions, BeautifulSoup, lxml/XPath, and CSS—detailing their installation, parsing behavior, advantages, drawbacks, and a performance comparison table to help developers choose the most suitable tool for their projects.
BeautifulSoupCSS selectorsPython
0 likes · 8 min read

This tutorial shows how to encode a JD.com search URL, fetch the page, and use XPath expressions in Python to reliably extract product names, links, images, and prices, offering a clearer alternative to regular‑expression or BeautifulSoup scraping.
PythonXPathweb scraping
0 likes · 4 min read

This article explains how an AI‑driven tool dramatically reduces XPath creation costs and improves stability for complex 3D‑canvas web applications by combining prompt engineering, HTML compression, self‑checking, retry mechanisms, and specialized handling for inputs and 3D model selectors.
AI automationUI testingXPath
0 likes · 10 min read

This article walks through a Python web‑scraping problem where an incorrect XPath selector caused unexpected results, demonstrates the faulty code, explains why the XPath was wrong, and provides a corrected script with proper headers to reliably extract the desired text.
XPathlxml
0 likes · 4 min read

This article demonstrates how to automate non‑select dropdowns that allow or disallow text input, locate elements with dynamic IDs or without unique attributes using XPath, overcome click‑obstruction with JavaScript, and upload files via pywinauto when standard input elements are unavailable, all with Python Selenium code examples.
File UploadPythonXPath
0 likes · 7 min read

This article explains how to use XPath in Python to skip the first table header row during web scraping, provides a concise code example, and discusses alternative approaches, helping readers efficiently extract desired list items from HTML structures.
Backend DevelopmentPythonXPath
0 likes · 3 min read

This article walks through using Python's pyquery library to fetch the names and URLs of popular songs from NetEase Cloud Music, compares it with previous xpath and BeautifulSoup approaches, provides complete runnable code, and highlights key challenges in selector usage.
PythonXPathnetease-music
0 likes · 5 min read

This article explains why XPath fails on NetEase Cloud Music pages, demonstrates a regex‑based Python solution with full source code, shows sample output, and outlines future tutorials using XPath, BeautifulSoup, and PyQuery to strengthen Python selector skills.
PythonXPathlxml
0 likes · 5 min read

Learn how to extract song names and URLs from NetEase Cloud Music's hot tracks page using Python's requests library and XPath selectors, including handling malformed HTML, code examples, and tips for replacing interfering tags to ensure reliable scraping.
PythonXPathhtml-parsing
0 likes · 5 min read

This article explains how to locate web elements in Playwright using CSS selectors, XPath, text and attribute-based strategies, and demonstrates dynamic waiting techniques and handling of dynamically changing or obscured elements for reliable UI automation testing.
CSS selectorsElement LocatingPlaywright
0 likes · 5 min read

Learn how to scrape Baidu search result titles and URLs using Python's requests library and lxml's xpath parsing, complete with a ready-to-run script, CSV output, and step-by-step explanation that builds on earlier regex and BeautifulSoup methods.
BaiduCSVPython
0 likes · 6 min read

This tutorial explains how to install Google Chrome and ChromeDriver on CentOS, use PHP‑WebDriver to scrape dynamic web pages, and introduces XPath syntax for locating elements, providing step‑by‑step commands, code samples, and execution results.
CentOSChromeDriverPHP
0 likes · 7 min read

This tutorial explains how to develop a Chrome extension that assists developers in locating web page elements and generating XPath expressions by outlining the plugin's structure, providing complete configuration files, JavaScript code, HTML/CSS UI, and step‑by‑step instructions for loading and testing the extension.
Chrome extensionJavaScriptPython
0 likes · 7 min read

This guide explains how to create a basic PHP web crawler by using cURL to fetch pages, DOMDocument and XPath to parse HTML, and then storing the extracted data, while also providing a complete example script and reminders about legal and ethical considerations.
Backend DevelopmentDOMDocumentPHP
0 likes · 3 min read

This article walks through a real‑world Python web‑scraping problem, demonstrating how to extract clean table data using pandas.read_html and an alternative XPath approach, complete with code snippets, output screenshots, and practical tips for robust data collection.
PandasPythonXPath
0 likes · 4 min read

This article walks through diagnosing and fixing an XPath extraction issue in a Python web scraper, showing the original faulty code, the corrected selector, and the final working script with sample outputs, while emphasizing best practices like setting request headers.
Backend DevelopmentXPathlxml
0 likes · 4 min read

An experienced Python developer walks through a real-world web‑scraping issue, showing how a faulty XPath selector caused empty results, then provides corrected code, execution screenshots, and best practices like adding request headers, helping readers quickly resolve similar problems.
Backend DevelopmentXPathlxml
0 likes · 4 min read

This article walks through a Python web‑scraping problem where price data extracted via XPath contains empty lines and unwanted characters, presenting multiple code solutions—including list comprehensions, map‑filter, and conditional printing—to clean and correctly display the product prices.
XPathlxmlrequests
0 likes · 6 min read

This article provides a detailed walkthrough of Scrapy, covering its event‑driven architecture, component interactions, XPath parsing fundamentals, installation steps, project creation, sample spider code, item pipelines, middleware customization, and essential configuration settings for effective web crawling in Python.
PipelineSpiderWeb Crawling
0 likes · 12 min read

This article explains how to use XPath in Python to bypass the first table header row during web scraping, provides a concise code example, and walks through the implementation steps so readers can adapt the technique to similar crawling tasks.
Backend DevelopmentXPathhtml-parsing
0 likes · 3 min read

Learn to extract the names and URLs of popular NetEase Cloud Music tracks using Python's pyquery library, with step‑by‑step code, selector tips, and comparisons to regex, xpath, and BeautifulSoup approaches, enabling reliable web‑scraping of music data.
XPathnetease-musicpyquery
0 likes · 5 min read

This tutorial explains how to fetch the names and URLs of hot songs from NetEase Cloud Music using Python, addressing XPath limitations by cleaning the HTML and parsing it with BeautifulSoup, and provides a complete, runnable script with example output.
BeautifulSoupXPathnetease-music
0 likes · 6 min read

Learn how to extract the names and URLs of popular NetEase Music tracks using Python's requests, lxml's XPath, and a simple HTML cleanup technique, with full code examples and tips for handling non‑standard HTML responses.
XPathlxmlnetease-music
0 likes · 6 min read

This article explains why XPath fails on NetEase Cloud Music's hot‑song page, presents a regular‑expression solution in Python, provides a complete script, and shows the resulting song names and URLs, while previewing upcoming tutorials using xpath, bs4, and pyquery.
BeautifulSoupXPathnetease-music
0 likes · 5 min read

Learn how to scrape Baidu Tieba thread titles and associated images using Python's requests library and XPath expressions, with a complete example script that handles malformed HTML, removes comment markers, and outputs title‑image pairs for further processing.
Baidu TiebaXPathlxml
0 likes · 5 min read

This article explains why XPath fails on Baidu Tieba pages, demonstrates how to extract thread titles and image URLs using Python's requests library combined with regular expressions, provides a complete runnable script, and shows the resulting output.
Baidu TiebaXPathregex
0 likes · 5 min read

This article demonstrates how to use Python's requests library and BeautifulSoup to crawl Baidu Tieba forums, extracting thread titles and associated images, explains why XPath may fail on irregular HTML, and provides a complete, runnable script with step-by-step implementation details.
Baidu TiebaBeautifulSoupXPath
0 likes · 5 min read

This article walks through a real-world Python web‑scraping issue where an extra slash in the URL caused failures, demonstrates debugging with try/except, shows a complete recipe‑crawling script using requests, lxml, and UserAgent, and explains proper URL joining with urllib.parse.urljoin.
XPathlxmlrequests
0 likes · 6 min read

This tutorial explains the fundamental concepts and parsing principles of XPath, shows how to set up the Python lxml environment, demonstrates instantiating etree objects, details XPath expression syntax, and provides multiple real‑world web‑scraping examples with complete code snippets.
PythonXPathhtml-parsing
0 likes · 9 min read

This tutorial shows how to crawl the China Meteorological Administration website to obtain today's weather and a six‑day forecast, handle Chinese encoding, extract data with XPath, and save the results into a CSV file using Python's requests and lxml libraries.
CSVPythonXPath
0 likes · 5 min read

This guide walks through using Python's requests and lxml libraries to scrape quotes, authors, and tags from quotes.toscrape.com, parse the HTML with XPath, and store the results in a CSV file, covering request headers, element inspection, and data extraction steps.
PythonXPathweb scraping
0 likes · 4 min read

This article introduces Spiderman, a Java-based open‑source web scraping tool that uses XPath and regular expressions, explains its micro‑kernel plugin architecture, and provides step‑by‑step instructions for configuring and running the tool to extract data from target web pages.
JavaSpidermanXPath
0 likes · 4 min read

This article walks you through using Python's Selenium library to programmatically retrieve Bilibili video series playlists, explains the required XPath selectors, provides a complete, runnable code example, and addresses common issues such as ChromeDriver compatibility.
XPathseleniumweb scraping
0 likes · 8 min read


21CTOJul 12, 2021 · Backend Development
This comprehensive guide introduces Scrapy's architecture, explains its core components and data flow, teaches XPath fundamentals, walks through installation, project creation, spider coding, item and pipeline definitions, middleware customization, pagination handling, and essential settings for effective Python web crawling.
CrawlerPythonXPath
0 likes · 14 min read

This tutorial explains how to use XPath in Python to extract free proxy IP addresses from kuaidaili.com, covering XPath basics, installing the XPath Helper Chrome extension, crafting extraction rules, and looping through pages to collect and display results.
BackendProxyTutorial
0 likes · 9 min read

This guide demonstrates how to use Python's Selenium library to log into a GPS web system, locate elements via name and XPath, handle iframes, click list items, and extract dynamic data, including detailed installation steps for Selenium and ChromeDriver.
PythonXPathdata-extraction
0 likes · 5 min read

This article explains the distinction between selenium‑webdriver and browser drivers, introduces HTML basics and element attributes, and details Selenium's element‑locating APIs with practical examples of XPath and CSS selector strategies for UI automation.
HTMLUI automationXPath
0 likes · 14 min read

This tutorial explains how to use Python's requests, lxml, and urllib libraries to search Baidu Tieba by keyword, bypass anti‑crawling measures, extract image and video URLs with XPath, and save the media files locally, complete with code examples and screenshots.
Baidu TiebaPythonXPath
0 likes · 9 min read

This article introduces the fundamentals of XPath syntax, common rules, and example expressions, then explains how to use the lxml library in Python for HTML/XML parsing, including practical tips and a complete code example for extracting links and text from a sample document.
PythonXMLXPath
0 likes · 6 min read

This tutorial shows how to configure Scrapy to start from a list page, extract every article link, follow pagination automatically, and parse each article using XPath/CSS selectors, with practical shell commands and visual examples.
CSS selectorsPaginationPython
0 likes · 6 min read

This article demonstrates how to locate and extract cover‑image URLs from a web page using Scrapy, explains handling absolute and relative URLs, shows the necessary XPath and meta‑passing code, and provides debugging tips to verify that the image URL is correctly transferred through the spider.
MetaPythonXPath
0 likes · 6 min read

This article continues a Scrapy tutorial by showing how to extract like counts, comment counts, and full article content using XPath selectors, regular expressions, and debugging techniques, providing step‑by‑step code examples and screenshots to help Python developers automate web data collection.
PythonXPathdata-extraction
0 likes · 6 min read

This tutorial shows how to apply XPath expressions within the Scrapy framework to extract titles, publication dates, tags, content, likes, favorites, and comments from a sample website, providing practical code snippets and tips for reliable web data collection.
PythonWeb CrawlingXPath
0 likes · 5 min read

This tutorial walks you through creating a Python web scraper that extracts images from a target site, handles anti‑scraping measures, saves them into categorized folders, and logs the download results, while explaining the required libraries, code structure, and best practices.
Image DownloadPythonXPath
0 likes · 6 min read

Learn how to build a Python web scraper that fetches upcoming movie details from Maoyan.com, covering environment setup, URL pagination, random User‑Agent handling, HTML parsing with XPath, data extraction, and result display, while highlighting best practices and limitations.
MaoyanPythonXPath
0 likes · 6 min read

This tutorial walks you through creating a Python web scraper that fetches image URLs from 51miz.com using requests and lxml, filters them with regular expressions, downloads the images, and demonstrates the complete workflow with code snippets and screenshots.
PythonXPathimage-downloader
0 likes · 5 min read

This tutorial explains how to build a Python web‑scraper that searches Baidu Tieba by keyword, bypasses anti‑crawling measures, extracts image and video URLs with XPath, and saves the media files locally, complete with code examples and setup instructions.
Baidu TiebaPythonXPath
0 likes · 8 min read

This comprehensive guide walks you through Python web crawling fundamentals, covering request handling, URL encoding, regular expressions, the requests library, XPath parsing, and lxml, complete with code snippets and practical examples to help you build effective scrapers.
XPathlxmlregex
0 likes · 13 min read

This guide walks you through the fundamentals of building a web crawler in Python, covering how to fetch pages with the Requests library, extract data using regular expressions and XPath, and provides practical code examples for each step.
Web CrawlingXPathdata-extraction
0 likes · 9 min read

This guide explains core XPath concepts—including node selection shortcuts, predicate filters, wildcard usage, multiple path unions, axis navigation, and built‑in functions—providing clear examples for effective XML/HTML data extraction.
SelectorsXMLXPath
0 likes · 6 min read

This tutorial walks you through using Scrapy's Xpath selectors to locate and extract titles, dates, comments, and content from web pages, demonstrating both manual and browser‑assisted methods, and shows how to integrate the expressions into your Scrapy spider for reliable data harvesting.
PythonXPathscrapy
0 likes · 6 min read

Learn how to disable robots.txt compliance in Scrapy, use the Scrapy shell for rapid URL debugging, and apply XPath selectors directly in the shell to efficiently extract data, dramatically speeding up development and avoiding repeated full-crawl executions.
PythonXPathdebugging
0 likes · 4 min read

This tutorial shows how to collect book metadata such as publisher, publication date, ISBN, price, rating and review count from Douban for a list of titles stored in Excel, using Python requests, lxml XPath parsing, pandas for merging and analysis, and visualizing the results with matplotlib.
PandasPythonXPath
0 likes · 14 min read

This article introduces XPath fundamentals, explains node hierarchies, common syntax, and practical examples with diagrams, enabling readers to confidently craft XPath expressions for extracting data from HTML and XML pages, and prepares them for applying XPath in Scrapy projects.
HTMLXMLXPath
0 likes · 6 min read

This guide explains XPath fundamentals, covering node selection syntax, predicates for filtering, wildcard usage, combining multiple paths, axis navigation, and built‑in functions, providing clear examples for each to help developers efficiently query XML or HTML documents.
SelectorsXMLXPath
0 likes · 6 min read

This guide explains the core XPath syntax—including node selectors, predicates, wildcards, multiple paths, axes, and built‑in functions—providing clear examples so you can efficiently locate and filter XML/HTML elements when scraping web pages.
SelectorsXMLXPath
0 likes · 6 min read

Learn to quickly build a Python web scraper using requests and Xpath to extract Douban movie 'Black Panther' short reviews, covering setup, HTTP request analysis, data parsing, storage with pandas, and best practices like polite crawling intervals, all demonstrated with concise 12-line code.
PandasXPathrequests
0 likes · 10 min read

This tutorial walks beginners through setting up a Python Scrapy project, writing a spider to fetch forum thread titles and contents, using XPath for parsing, and enhancing the crawler with pipelines, middleware, and common settings for robust web scraping.
PipelineXPathmiddleware
0 likes · 13 min read

This tutorial walks you through using Python's requests and XPath to scrape short comments from Douban's "Black Panther" page, covering tool setup, HTTP request analysis, data extraction, and saving results to CSV in just a dozen lines of code.
PandasXPathrequests
0 likes · 9 min read

This article demonstrates a compact, runnable Python 3 scraper that fetches job listings from Lagou.com based on a keyword, filters by city and salary, extracts detailed job requirements using XPath, and applies Baidu's free NLP service for word segmentation and part‑of‑speech tagging to reveal key skill terms.
Baidu AIJob DataLagou
0 likes · 15 min read

This tutorial shows how to use Python's urllib and XPath to accurately extract product details such as name, link, image, and price from JD.com search results, offering a clearer alternative to regular expressions and BeautifulSoup.
JD.comPythonXPath
0 likes · 4 min read

This guide explains how to use XPath efficiently for web scraping, covering node selection, axes, functions, numeric comparisons, and advanced combinations, while emphasizing concise and readable expressions to improve performance and maintainability.
PythonXMLXPath
0 likes · 5 min read

Learn how to set up a Scrapy project to crawl comprehensive car brand, series, and model data from Autohome, covering environment preparation, project initialization, spider and pipeline creation, CSV output, rate limiting, and useful debugging tips.
AutohomeCSV exportCar Data
0 likes · 10 min read

This tutorial demonstrates how to use Python, Selenium, and XPath to crawl Huaban's dynamic image boards, download high‑resolution pictures, organize them into folders, and handle varying page structures, providing a practical example of web‑scraping automation.
Image DownloadPythonXPath
0 likes · 5 min read

This tutorial walks you through creating a Scrapy project, defining items, writing a spider, and exporting data to crawl the DMOZ website, covering command‑line setup, XPath extraction, handling encoding errors, and using pipelines for storage.
Item PipelinePythonWeb Crawling
0 likes · 11 min read

This tutorial walks through preparing the PMD environment, exploring its directory structure, and step‑by‑step instructions for implementing a custom rule—such as enforcing braces on while loops—using both Java code and XPath expressions, including packaging and execution details.
JavaPMDXPath
0 likes · 10 min read

The article explains why simple String comparison is insufficient for XML validation, introduces the Java‑based XMLUnit library, and demonstrates how to use it with JUnit/TestNG, Diff, and XPath to reliably test XML structure, content, and specific values.
JavaXML TestingXMLUnit
0 likes · 12 min read
