Master XPath and lxml: A Complete Guide to XML Parsing in Python

XPath Overview

XPath is a language for locating information in XML documents. It uses path expressions to navigate the document tree, includes a standard function library, is a core component of XSLT, and is defined as a W3C standard.

XPath Concepts

XPath defines seven node types: element, attribute, text, namespace, processing instruction, comment, and the document (root) node. Node relationships include parent, child, sibling, ancestor, and descendant.

XPath Syntax

Common path expressions include: nodename – selects all child nodes of the current node. / – selects from the root node. // – selects nodes anywhere in the document, regardless of location. . – selects the current node. .. – selects the parent of the current node. @ – selects an attribute.

Predicates

Predicates, placed in square brackets, filter nodes by position or value, e.g., /bookstore/book[1] selects the first book element, and //title[@lang='eng'] selects title elements with a lang attribute equal to "eng".

Wildcard Selection

Wildcards such as *, @*, and node() match any element, any attribute, or any node type, respectively.

Combining Paths

The union operator | allows selection of multiple paths, e.g., //book/title | //book/price selects both title and price elements.

lxml Library

lxml is a high‑performance Python library for parsing XML and HTML, offering fast XPath support. Install it via pip install lxml or easy_install lxml.

Parsing Offline HTML

from lxml import etree
html = etree.parse('xx.html', etree.HTMLParser())
links = html.xpath('//*[@id="s_xmancard_news"]/div/div[2]/div/div[1]/h2/a[1]/@href')
print(links)

Parsing Online HTML

from lxml import etree
import requests
resp = requests.get('https://www.baidu.com')
html = etree.HTML(resp.text)
links = html.xpath('//*[@id="s_xmancard_news"]/div/div[2]/div/div[1]/h2/a[1]/@href')
print(links)

To retrieve an element’s text and attribute, you can use XPath directly:

texts = html.xpath('//*[@id="s_xmancard_news"]/div/div[2]/div/div[1]/h2/a[1]/text()')
hrefs = html.xpath('//*[@id="s_xmancard_news"]/div/div[2]/div/div[1]/h2/a[1]/@href')

Or fetch the element first and then access its properties:

elem = html.xpath('//*[@id="s_xmancard_news"]/div/div[2]/div/div[1]/h2/a[1]')[0]
text = elem.text
href = elem.attrib.get('href')

Below is a visual illustration of extracting elements:

Advanced lxml XPath Examples

# Select all nodes
result = html.xpath('//*')
# Select all <li> elements
result = html.xpath('//li')
# Get text of <a> inside <li>
result = html.xpath('//li/a/text()')
# Filter by attribute value
result = html.xpath('//li[@class="ni"]')
# Use functions like last(), position(), contains()
result = html.xpath('//li[last()]/a/text()')
result = html.xpath('//li[position()<3]/a/text()')
result = html.xpath('//li[contains(@class,"li")]/a/text()')

For more details, refer to the W3C XPath specification and the lxml documentation.