Master XPath: Essential Node Selection, Predicates, and Functions for Web Scraping
This guide explains the core XPath syntax—including node selectors, predicates, wildcards, multiple paths, axes, and built‑in functions—providing clear examples so you can efficiently locate and filter XML/HTML elements when scraping web pages.
Node Selection
XPath provides several shortcuts for selecting nodes: nodename – selects all child nodes of the given name, e.g.,
xpath('//div')/– selects from the document root, e.g.,
xpath('/div')//– selects all matching nodes regardless of depth, e.g.,
xpath('//div').– selects the current node, e.g.,
xpath('./div')..– selects the parent of the current node, e.g.,
xpath('..')@– selects an attribute, e.g.,
xpath('//@class')Predicates
Predicates, placed inside square brackets, filter nodes based on position, attribute values, or other conditions. xpath('/body/div[1]') – first div under
bodyxpath('/body/div[last()]')– last div under
bodyxpath('/body/div[last()-1]')– second‑last
divxpath('/body/div[position()<3]')– first two div elements xpath('/body/div[@class]') – div elements that have a class attribute xpath('/body/div[@class="main"]') – div with
class="main"xpath('/body/div[price>35.00]')– div whose price child is greater than 35
Wildcards
Wildcards let you match unknown elements: xpath('/div/*') – all child nodes of a
divxpath('/div[@*]')– all div elements that have any attribute
Multiple Paths
Use the pipe operator | to combine paths: xpath('//div|//table') – selects every div and table node
XPath Axes
Axes define node sets relative to the current node: ancestor – xpath('./ancestor::*') selects all ancestors (parents, grandparents) ancestor-or-self – xpath('./ancestor-or-self::*') selects ancestors and the node itself attribute – xpath('./attribute::*') selects all attributes of the node child – xpath('./child::*') selects all child nodes descendant – xpath('./descendant::*') selects all descendants (children, grandchildren, …) following – xpath('./following::*') selects nodes after the current node’s end tag following-sibling – xpath('./following-sibling::*') selects following sibling nodes parent – xpath('./parent::*') selects the parent node preceding – xpath('./preceding::*') selects nodes before the current node’s start tag preceding-sibling – xpath('./preceding-sibling::*') selects preceding sibling nodes self – xpath('./self::*') selects the current node itself
Functions
Built‑in functions enable more flexible queries: starts-with – xpath('//div[starts-with(@id,"ma")]') selects div elements whose id starts with “ma” contains – xpath('//div[contains(@id,"ma")]') selects div elements whose id contains “ma” and – xpath('//div[contains(@id,"ma") and contains(@id,"in")]') selects div elements whose id contains both “ma” and “in” text() – xpath('//div[contains(text(),"ma")]') selects div elements whose text contains “ma”
For more details, see the Scrapy XPath documentation: http://doc.scrapy.org/en/0.14/topics/selectors.html
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
