How to Scrape Baidu Keywords and Links with Python & BeautifulSoup

1. Introduction

In a recent Python community share, a member posted a script that extracts Baidu search keywords and their corresponding links. The original version used regular expressions; this tutorial replaces that approach with BeautifulSoup for more reliable HTML parsing.

2. Implementation

Below is the complete Python code that performs the scraping, processes pagination, and writes the results to a CSV file.

# -*- coding: utf-8 -*-
# @Time    : 2022/4/20  18:24
# @Author  : 皮皮：Python共享之家
# @File    : demo.py

import requests
from bs4 import BeautifulSoup
import time
import pandas as pd

# From the element analysis we know Baidu returns an encrypted URL

def convert_url(url):
    resp = requests.get(url=url,
                        headers=headers,
                        allow_redirects=False)
    return resp.headers['Location']

# Get URLs

def get_url(wd, num):
    s = requests.session()
    total_title = []
    total_url = []
    total_info = []
    # Page 1 is numbers less than 10, 10 is page 2, 20 page 3, etc.
    num = num * 10 - 10
    for i in range(-10, num, 10):
        url = 'https://www.baidu.com/s'  # Example: https://www.baidu.com/s?wd=python&pn=10
        params = {
            "wd": wd,
            "pn": i,
        }
        r = s.get(url=url, headers=headers, params=params)
        print("返回状态码:", r.status_code)  # 200 means normal page access
        soup = BeautifulSoup(r.text, 'lxml')
        for so in soup.select('#content_left .t a'):
            # g_url = convert_url(so.get('href'))  # Convert to real URL if needed
            g_url = so.get('href')
            g_title = so.get_text().replace('
', '').strip()
            print(g_title, g_url)
            total_title += [g_title]
            total_url += [g_url]
        time.sleep(1 + (i / 10))
        print("当前页码：", (i + 10) / 10 + 1)
    try:
        total_info = zip(total_title, total_url)
        df = pd.DataFrame(data=total_info, columns=['标题', 'Url'])
        df.to_csv('./web_data.csv', index=False, encoding='utf_8_sig')
        print("保存成功")
    except:
        return 'FALSE'

if __name__ == '__main__':
    while True:  # Loop
        headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0",
            "Host": "www.baidu.com",
        }
        wd = input("输入搜索内容：")
        num = int(input("输入页数："))
        get_url(wd, num)

Running the script produces a console output similar to the screenshot below.

The program also automatically generates a CSV file named web_data.csv containing the collected titles and URLs, as shown in the second image.

3. Conclusion

This tutorial demonstrates a practical way to scrape Baidu search results using Python, requests, and BeautifulSoup, saving the extracted data to a CSV file. The next article will explore extracting the same information with XPath, encouraging readers to try it themselves.