Master XPath: Navigate HTML & XML Like a Pro for Web Scraping
This article introduces XPath fundamentals, explains node hierarchies, common syntax, and practical examples with diagrams, enabling readers to confidently craft XPath expressions for extracting data from HTML and XML pages, and prepares them for applying XPath in Scrapy projects.
What is XPath?
XPath allows locating elements in HTML or XML using a path‑like syntax, similar to file system paths, and provides a rich set of functions.
Node hierarchy
HTML/XML nodes have parent, child, sibling, ancestor, and descendant relationships. For example, <head> is a node; <body> is the parent of <nav>, and <nav> is a child of <body>. Sibling nodes share the same level, while ancestors are higher‑level nodes and descendants are lower‑level nodes.
Common XPath syntax
//@class– select all nodes with a class attribute. /article – select the root article element. //div – select all div elements. article – select all child nodes of an article element. article/a – select a elements that are children of article. article//div – select div descendants of article. /article/div[1] – select the first div child of article. /article/div[last()] – select the last div child of article. /div/* – select all direct child nodes of a div. //* – select all elements. //div/a | //div/p – select a or p elements under any div.
The single slash / selects immediate child nodes, while double slash // selects descendants at any depth. The @ symbol filters by attribute, such as @class.
Practical example
An example HTML snippet shows a class="grid-5" attribute; XPath can target such elements using expressions like //div[@class='grid-5'].
Further XPath expressions
//div[@lang]– select div elements with a lang attribute. //div[@lang='eng'] – select div elements where lang equals eng. /article/div[1] – first div under article. /article/div[last()] – last div under article. /div/* – all child nodes of a div. //* – all elements. //div/a | //div/p – a and p elements under any div.
With these basics, you can write XPath expressions to extract target data from web pages. The next article will demonstrate using XPath within a Scrapy spider.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
Python Crawling & Data Mining
Life's short, I code in Python. This channel shares Python web crawling, data mining, analysis, processing, visualization, automated testing, DevOps, big data, AI, cloud computing, machine learning tools, resources, news, technical articles, tutorial videos and learning materials. Join us!
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.