Master XPath for Precise JD.com Product Scraping with Python

Previously the author demonstrated crawling JD.com product data with Python regular expressions and BeautifulSoup; this article introduces a more precise method using XPath.

HTML pages consist of nested tags forming a tree, and XPath selects nodes via path expressions.

To scrape JD.com, search for a keyword (e.g., "dog food"), which appears in the URL as the keyword parameter. Encode the keyword (UTF‑8) using urllib.parse.quote, request the page, and then parse the HTML.

The desired product information resides inside <li data-sku="..." class="gl-item"> elements. By iterating over these nodes, you can extract each item's name, link, image, and price.

URL encoding converts characters to %xx form; Python’s urllib.parse.quote handles this conversion.

Although online tools can generate XPath expressions, they often produce unreadable strings; writing and testing your own XPath is recommended.

Example code:

items = selector.xpath('//li[@class="gl-item"]')

Loop through the items to retrieve details:

title = selector.xpath('//div[@class="p-img"]/a')[i].get('title')

The final output shows the extracted product information, confirming that XPath provides a simpler and more reliable approach than regular expressions for JD.com scraping.