How to Build a Zhihu Beauty Photo Scraper with Python and Baidu AI Face Detection

1 Data Source

Images from answers to all questions under Zhihu topic "美女".

2 Crawling Tool

Python 3 with third‑party libraries Requests, lxml, and AipFace; the script exceeds 100 lines.

3 Required Environment

Mac, Linux, or Windows (Linux untested). Windows may need filename character filtering. No Zhihu login required; a Baidu Cloud account is needed for face detection.

4 Face Detection Library

AipFace, a Python SDK from Baidu Cloud AI platform, provides HTTP‑based face detection and is free to use.

5 Filtering Conditions

Discard images without faces (e.g., landscapes, body‑only shots).

Discard non‑female images (male images are mostly celebrities; gender detection may be inaccurate).

Discard non‑real persons such as anime characters (AipFace human confidence < 0.6).

Discard images with low beauty scores (AipFace beauty < 45) to save storage; note the score is subjective.

6 Implementation Logic

Use Requests to fetch a list of discussions under the "美女" topic.

Parse each discussion’s HTML with lxml to extract all img tag src attributes.

Download each image via Requests (ignore animated images).

Send the image to AipFace for face detection.

Apply the filtering conditions from section 5.

Save the filtered images to the local file system with filenames composed of beauty score, author, question title, and an index.

Repeat from step 1.

7 Crawl Results

The images are stored in a folder (e.g., "angelababy 实力出境"). The highest beauty score observed (excluding the celebrity Angelababy) is 88.

8 Code

The full script exceeds a hundred lines; the author provides it via a WeChat public account keyword "知乎爬虫".

9 Run Preparation

Install Python 3.

Install requests, lxml, and baidu-aip via pip (single command).

Apply for a free Baidu Cloud face detection service.

Fill in AppID, APIKey, and SecretKey in the code; optional custom settings include image storage directory, beauty threshold, and face confidence.

{
    "error": {
        "message": "ZERR_NO_AUTH_TOKEN",
        "code": 100,
        "name": "AuthenticationInvalidRequest"
    }
}

10 Conclusion

The face detection may filter out some adult content; Baidu also offers a pornography detection API.

If you prefer not to use Baidu Cloud, comment out the face detection part and run the crawler alone.

The detection step can be replaced with other providers or local models; Baidu is used because it is free.

Thousands of photos were collected with decent results; feel free to run the code and experiment.

The article demonstrates a basic crawler plus data filtering; it can be easily adapted to other topics, fields, or additional filters, and extended with other machine‑learning APIs.

Remember that beauty scores and gender filters may produce false cases; do not treat them as definitive.