How to Scrape High‑Resolution Images from Huaban with Selenium & XPath

1. Introduction

The author discovered the image‑sharing site Huaban and decided to use Selenium with XPath to crawl the "beauty" board, saving images into folders named after each board. Because the site loads content dynamically, scrolling simulation is needed to retrieve more images. The first version captured about 500 images from 19 boards.

2. Runtime Environment

IDE: PyCharm

Python 3.6

lxml 3.7.2

Selenium 3.4.0

requests 2.12.4

3. Example Analysis

The crawling process starts by opening http://huaban.com/boards/favorite/beauty , extracting the URLs of all image boards, and then visiting each board page to collect the images. Screenshots illustrate the page structure and the extracted URLs.

After the first pass, the images obtained were low‑resolution (236×354). To get higher‑resolution pictures, a second version navigates into each thumbnail’s detail page and downloads the original image.

4. Practical Code

1. Import the required modules.

2. Configure the WebDriver (Firefox, PhantomJS, etc.). Options such as '--load-images=false' and '--disk-cache=true' speed up crawling. WebDriverWait sets a maximum wait time of 10 seconds, and set_window_size defines the browser viewport.

3. Define a parser(url, param) function that parses a page given its URL and a visible element to wait for (e.g., a button or image).

4. Parse the main board page to obtain each board’s URL and name. The board name is used to create a local folder; illegal characters (e.g., '*') are filtered out.

5. For each board, the script first collects thumbnail URLs, then visits each thumbnail’s detail page to fetch the high‑resolution image URL. Because different boards use different DOM structures, the code handles both cases and merges the resulting URL lists. img_url += img_url2 Images are saved locally with: filename = 'image\{}\'.format(fileName) + str(i) + '.jpg' The files are stored in an image directory alongside the script, organized by board name.

5. Conclusion

This exercise reinforced the use of Selenium and XPath for web crawling, highlighted challenges in dynamic page analysis, and resulted in the successful download of over 500 high‑quality images.