Mastering XPath: Powerful Techniques for Precise Web Scraping

Experiment Environment

Python with lxml.etree.

XPath Basics

Use .// to match all nodes under a node, // to select from the whole document, and . for the current node.

Attribute Selection

Retrieve attribute values with //@lang.

Multiple Paths

Combine expressions using the | operator; each expression works independently.

Axes

child : select all child elements of the current node.

attribute : select all attributes of the current node.

ancestor / ancestor-or-self : select ancestor nodes, optionally including the current node.

descendant / descendant-or-self : select descendant nodes, optionally including the current node.

following : select all nodes after the end tag of the current node.

namespace : select namespace nodes of the current node.

parent : select the parent node.

preceding : select all nodes before the start tag of the current node.

preceding-sibling : select preceding sibling nodes.

self : select the current node itself.

Position and Conditions

Use position() for node position and predicates for filtering.

Functions

count() : count nodes.

concat() : concatenate strings.

string() : get the string value of a node.

local-name() : get the node name.

contains() : test if one string contains another.

not() : logical negation.

string-length() : length of a string.

Advanced Combinations

Examples combine axes, functions, numeric comparisons ( <, div), and modulus ( position() mod 2) to achieve complex selections such as selecting every second row or filtering by attribute values.

Serializing Nodes

Convert a node set back to a string using the string() function.

Conclusion: XPath is fast, but prefer concise, efficient expressions; overly obscure tricks reduce readability and performance.