Master XPath: Essential Node Selection, Predicates, Axes, and Functions

1. Selecting Nodes

Common XPath expressions for node selection include: nodename – selects all child nodes of the given node name, e.g., xpath('//div') selects all div children. / – selects from the root node, e.g., xpath('/div') selects the div node at the document root. // – selects nodes anywhere in the document regardless of position, e.g., xpath('//div') selects all div elements. . – selects the current node, e.g., xpath('./div') selects div children of the current node. .. – selects the parent of the current node, e.g., xpath('..') moves up one level. @ – selects attributes, e.g., xpath('//@class') selects all class attributes.

2. Predicates

Predicates are placed inside square brackets to filter nodes by position, attribute values, or other conditions. xpath('/body/div[1]') – first div under body. xpath('/body/div[last()]') – last div under body. xpath('/body/div[last()-1]') – second‑last div under body. xpath('/body/div[position()<3]') – the first two div elements. xpath('/body/div[@class]') – div elements that have a class attribute. xpath('/body/div[@class="main"]') – div with class="main". xpath('/body/div[price>35.00]') – div whose price child value exceeds 35.

3. Wildcards

Wildcard expressions allow selection of unknown elements: xpath('/div/*') – all child nodes of any div. xpath('/div[@*]') – all div elements that have any attribute.

4. Selecting Multiple Paths

The pipe operator | combines paths, e.g., xpath('//div|//table') selects all div and table nodes.

5. XPath Axes

Axes define node sets relative to the current node: ancestor – xpath('./ancestor::*') selects all ancestor nodes. ancestor-or-self – xpath('./ancestor-or-self::*') selects ancestors and the node itself. attribute – xpath('./attribute::*') selects all attributes of the current node. child – xpath('./child::*') selects all child nodes. descendant – xpath('./descendant::*') selects all descendant nodes. following – xpath('./following::*') selects nodes after the current node’s end tag. following-sibling – xpath('./following-sibling::*') selects following sibling nodes. parent – xpath('./parent::*') selects the parent node. preceding – xpath('./preceding::*') selects nodes before the current node’s start tag. preceding-sibling – xpath('./preceding-sibling::*') selects preceding sibling nodes. self – xpath('./self::*') selects the current node itself.

6. Functions

Functions enable more flexible matching: starts-with – xpath('//div[starts-with(@id,"ma")]') selects div elements whose id starts with “ma”. contains – xpath('//div[contains(@id,"ma")]') selects div elements whose id contains “ma”. and – xpath('//div[contains(@id,"ma") and contains(@id,"in")]') selects div elements whose id contains both “ma” and “in”. text() – xpath('//div[contains(text(),"ma")]') selects div elements whose text contains “ma”.

For further reference, see the Scrapy XPath documentation.