How to Accurately Scrape JD.com Product Data with BeautifulSoup

Yesterday the author used Python regular expressions to crawl JD.com product information, but the code was long and cumbersome. Today, BeautifulSoup is demonstrated to achieve precise matching of JD product data.

HTML files consist of tags organized in a tree structure; BeautifulSoup parses, traverses, and maintains this tag tree.

First, visit JD.com and enter a product keyword (e.g., "dog food"). The request URL becomes

https://search.jd.com/Search?keyword=%E7%8B%97%E7%B2%AE&enc=utf-8

, where the keyword parameter represents the search term. After encoding the keyword, send the request, receive the response, and use BeautifulSoup selectors for data extraction.

The relevant product information resides within <li data-sku="*****" class="gl-item"> tags, so we peel the HTML layers like an onion to retrieve the desired fields.

URL encoding converts characters to %xx form, typically using UTF‑8. Python's urllib.parse.quote method handles this encoding.

Using BeautifulSoup, we extract the product name, link, image, and price. The extraction code (shown in the image) demonstrates these steps.

Note that some image URLs may be empty; using img.get('src') avoids errors, returning None when absent, or handling exceptions with try/except. This is a useful BeautifulSoup tip.

The final output shows the extracted product details, confirming that BeautifulSoup provides a simpler and more reliable approach than regular expressions for JD.com data scraping.