Crawl Zhihu’s “Beautiful Women” Images and Filter by AI Face Scores in Python

1. Data Source

Images from answers to questions under Zhihu topic “美女”.

2. Crawling Tool

Python 3 with third‑party libraries Requests, lxml, and Baidu AipFace; the script is about 100+ lines.

3. Required Environment

Works on Mac, Linux, or Windows (Windows may need filename character filtering). No Zhihu login required; a Baidu Cloud account is needed for the face detection service.

4. Face Detection Library

AipFace, a free Python SDK from Baidu Cloud AI platform, provides HTTP‑based face detection.

5. Filtering Conditions

Discard images without faces (e.g., landscapes).

Keep only female faces (male images are mostly celebrities; AipFace gender may be inaccurate).

Exclude non‑real persons such as anime characters (Human confidence < 0.6).

Exclude low beauty scores (beauty < 45); note that scores are not objective.

6. Implementation Logic

Use Requests to fetch a list of discussions under “美女”.

Parse each discussion’s HTML with lxml to extract all img tag src attributes.

Download each image (static images only) via Requests.

Send the image to AipFace for detection.

Apply the filtering conditions from section 5.

Save the remaining images locally with filenames composed of beauty score, author, question title, and an index.

Repeat from step 1 until the desired amount is collected.

7. Crawl Results

Images are stored in a folder (e.g., “angelababy”). The highest beauty score observed (aside from the celebrity “baby”) is 88.

8. Code

The full script exceeds 100 lines; the author provides it via a WeChat keyword request (“知乎爬虫”).

9. Preparation for Running

Install Python 3.

Install requests, lxml, and baidu-aip via pip.

Apply for a free Baidu Cloud face detection service (AI → Face Recognition).

10. Common Error Example

{
    "error": {
        "message": "ZERR_NO_AUTH_TOKEN",
        "code": 100,
        "name": "AuthenticationInvalidRequest"
    }
}

11. Conclusion

Face detection may filter out some “benefit” images; Baidu also offers a pornography detection API.

If you prefer not to use Baidu, comment out the face‑detection part and run the crawler alone.

The detection step can be replaced by other providers or local models.

Thousands of images were collected with satisfactory quality; the script is easily adaptable for different topics, fields, or additional filters.

Remember that beauty scores and gender filters are not reliable; treat them as rough references only.