Mastering XPath: Key Expressions, Axes & Functions for Precise Extraction

1. Node Selection

xpath('//div')

– selects all child nodes of the div element (nodename selector). xpath('/div') – selects the div node from the root. xpath('//div') – selects all div nodes regardless of position. xpath('./div') – selects div nodes that are direct children of the current node. xpath('..') – moves to the parent node of the current node. xpath('//@class') – selects all class attributes in the document.

2. Predicates

xpath('/body/div[1]')

– selects the first div under body. xpath('/body/div[last()]') – selects the last div under body. xpath('/body/div[last()-1]') – selects the second‑last div under body. xpath('/body/div[position()<3]') – selects the first two div nodes under body. xpath('/body/div[@class]') – selects div nodes that have a class attribute. xpath('/body/div[@class="main"]') – selects div nodes whose class equals "main". xpath('/body/div[price>35.00]') – selects div nodes where the price element is greater than 35.

3. Wildcards

xpath('/div/*')

– selects all child nodes of a div. xpath('/div[@*]') – selects all div elements that have any attribute.

4. Multiple Paths

xpath('//div|//table')

– selects every div and table node in the document.

5. XPath Axes

ancestor : xpath('./ancestor::*') – all ancestor nodes (parent, grand‑parent, etc.).

ancestor-or-self : xpath('./ancestor-or-self::*') – ancestors plus the current node.

attribute : xpath('./attribute::*') – all attributes of the current node.

child : xpath('./child::*') – all direct child nodes.

descendant : xpath('./descendant::*') – all descendant nodes (children, grandchildren, …).

following : xpath('./following::*') – all nodes after the closing tag of the current node.

following-sibling : xpath('./following-sibling::*') – sibling nodes that follow the current node.

parent : xpath('./parent::*') – the immediate parent node.

preceding : xpath('./preceding::*') – all nodes before the opening tag of the current node.

preceding-sibling : xpath('./preceding-sibling::*') – sibling nodes that precede the current node.

self : xpath('./self::*') – the current node itself.

6. Functions for Flexible Matching

starts-with

: xpath('//div[starts-with(@id,"ma")]') – selects div elements whose id starts with "ma". contains: xpath('//div[contains(@id,"ma")]') – selects div elements whose id contains "ma". and: xpath('//div[contains(@id,"ma") and contains(@id,"in")]') – selects div elements whose id contains both "ma" and "in". text(): xpath('//div[contains(text(),"ma")]') – selects div elements whose text content includes "ma".

For further reference, see the Scrapy XPath documentation: http://doc.scrapy.org/en/0.14/topics/selectors.html .