Master XPath: Essential Node Selection, Predicates, Axes, and Functions

Node Selection

nodename : selects all child nodes of the specified node, e.g., xpath('//div').

/ : selects from the root node, e.g., xpath('/div').

// : selects all matching nodes anywhere in the document, e.g., xpath('//div').

. : selects the current node, e.g., xpath('./div').

.. : selects the parent of the current node, e.g., xpath('..').

@ : selects an attribute, e.g., xpath('//@class').

Predicates

xpath('/body/div[1]')

– selects the first div under body. xpath('/body/div[last()]') – selects the last div under body. xpath('/body/div[last()-1]') – selects the second‑last div under body. xpath('/body/div[position()<3]') – selects the first two div elements. xpath('/body/div[@class]') – selects div elements that have a class attribute. xpath('/body/div[@class="main"]') – selects div elements whose class equals “main”. xpath('/body/div[price>35.00]') – selects div elements whose price child value is greater than 35.

Wildcards

xpath('/div/*')

– selects all child nodes of a div. xpath('/div[@*]') – selects all div elements that have any attribute.

Multiple Paths

xpath('//div|//table')

– selects all div and table nodes.

XPath Axes

ancestor : xpath('./ancestor::*') – all ancestor nodes of the current node.

ancestor-or-self : xpath('./ancestor-or-self::*') – ancestors plus the node itself.

attribute : xpath('./attribute::*') – all attributes of the current node.

child : xpath('./child::*') – all child nodes.

descendant : xpath('./descendant::*') – all descendant nodes.

following : xpath('./following::*') – nodes after the current node’s end tag.

following-sibling : xpath('./following-sibling::*') – sibling nodes after the current node.

parent : xpath('./parent::*') – the parent node.

preceding : xpath('./preceding::*') – nodes before the current node’s start tag.

preceding-sibling : xpath('./preceding-sibling::*') – sibling nodes before the current node.

self : xpath('./self::*') – the current node itself.

Functions

starts-with : xpath('//div[starts-with(@id,"ma")]') – selects div elements whose id starts with “ma”.

contains : xpath('//div[contains(@id,"ma")]') – selects div elements whose id contains “ma”.

and : xpath('//div[contains(@id,"ma") and contains(@id,"in")]') – selects div elements whose id contains both “ma” and “in”.

text() : xpath('//div[contains(text(),"ma")]') – selects div elements whose text contains “ma”.

For more details, see the Scrapy XPath documentation: http://doc.scrapy.org/en/0.14/topics/selectors.html