Master XPath: Essential Node Selection, Predicates, and Functions for Web Scraping

Node Selection

XPath provides several shortcuts for selecting nodes: nodename – selects all child nodes of the given name, e.g.,

xpath('//div')

/

– selects from the document root, e.g.,

xpath('/div')

//

– selects all matching nodes regardless of depth, e.g.,

xpath('//div')

.

– selects the current node, e.g.,

xpath('./div')

..

– selects the parent of the current node, e.g.,

xpath('..')

@

– selects an attribute, e.g.,

xpath('//@class')

Predicates

Predicates, placed inside square brackets, filter nodes based on position, attribute values, or other conditions. xpath('/body/div[1]') – first div under

body

xpath('/body/div[last()]')

– last div under

body

xpath('/body/div[last()-1]')

– second‑last

div

xpath('/body/div[position()<3]')

– first two div elements xpath('/body/div[@class]') – div elements that have a class attribute xpath('/body/div[@class="main"]') – div with

class="main"

xpath('/body/div[price>35.00]')

– div whose price child is greater than 35

Wildcards

Wildcards let you match unknown elements: xpath('/div/*') – all child nodes of a

div

xpath('/div[@*]')

– all div elements that have any attribute

Multiple Paths

Use the pipe operator | to combine paths: xpath('//div|//table') – selects every div and table node

XPath Axes

Axes define node sets relative to the current node: ancestor – xpath('./ancestor::*') selects all ancestors (parents, grandparents) ancestor-or-self – xpath('./ancestor-or-self::*') selects ancestors and the node itself attribute – xpath('./attribute::*') selects all attributes of the node child – xpath('./child::*') selects all child nodes descendant – xpath('./descendant::*') selects all descendants (children, grandchildren, …) following – xpath('./following::*') selects nodes after the current node’s end tag following-sibling – xpath('./following-sibling::*') selects following sibling nodes parent – xpath('./parent::*') selects the parent node preceding – xpath('./preceding::*') selects nodes before the current node’s start tag preceding-sibling – xpath('./preceding-sibling::*') selects preceding sibling nodes self – xpath('./self::*') selects the current node itself

Functions

Built‑in functions enable more flexible queries: starts-with – xpath('//div[starts-with(@id,"ma")]') selects div elements whose id starts with “ma” contains – xpath('//div[contains(@id,"ma")]') selects div elements whose id contains “ma” and – xpath('//div[contains(@id,"ma") and contains(@id,"in")]') selects div elements whose id contains both “ma” and “in” text() – xpath('//div[contains(text(),"ma")]') selects div elements whose text contains “ma”

For more details, see the Scrapy XPath documentation: http://doc.scrapy.org/en/0.14/topics/selectors.html