How to Fix Common XPath Errors in Python Web Scraping – A Step-by-Step Guide

Introduction

In a Python community a member asked about a web‑scraping selector issue; the original XPath did not return the expected results.

Problem

The original code used the following XPath (shown in the image) and printed the result, but the output was incorrect.

from lxml import etree
import requests
url = "http://www.xiaohua.com/duanzi/"

resp = requests.get(url)
html = etree.HTML(resp.text)

print('*---*'*20)

result = html.xpath("/html/body/div[@class='main']/div[@class='content']/div[@class='grid clearfix']/div[@class='content-left']/div[@class='one-cont'][*]/p[@class='fonts']")
print(type(result))
print(result)
print('*-*'*20)
b = 0
for i in result:
    b += 1
    print(i,len(result))
    print(b,etree.tostring(i).decode('utf-8'))
    if b > 1:
        break

The issue was identified as an incorrect XPath expression.

Solution

A community member provided a corrected XPath and updated code (illustrated in the image). After running the revised script the desired joke text is extracted correctly.

Key points include adding appropriate request headers to avoid being blocked.

Conclusion

The article demonstrates how to diagnose and fix XPath problems in Python web scraping, offering a concrete example and best‑practice tips for reliable data extraction.