How I Scraped 4,400 Taobao 'Big Pants' Listings and Uncovered Market Insights

In the scorching summer, the author J decided to use Python Selenium to scrape Taobao for "big pants" (men's shorts) and collect 4,403 product records for analysis.

Data Acquisition

Taobao loads content dynamically via AJAX, so Selenium was employed to automate the browser, log in via QR code, and retrieve product name, price, number of buyers, shop name, and shipping address, saving the results to big_pants.xlsx. The Chrome driver version was matched to the browser.

def main():
    browser.get('https://www.taobao.com/')
    page = search_product(key_word)
    print(page)
    get_data()
    page_num = 1
    while int(page) != page_num:
        print("-" * 100)
        print("正在爬取第{}页大裤衩数据".format(page_num + 1))
        browser.get('https://s.taobao.com/search?q={}&s={}'.format(key_word, page_num*44))
        browser.implicitly_wait(10)
        get_data()
        page_num += 1
    print("大裤衩数据抓取完成")
if __name__ == '__main__':
    key_word = "大裤衩 男"
    browser = webdriver.Chrome("./chromedriver")
    main()

Data Cleaning

After scraping, the raw data were cleaned in several steps:

Added column names.

Removed duplicate records.

Handled missing values.

Processed the price field.

Standardized the shipping address field.

Converted the buyer count field to numeric.

Other miscellaneous cleaning.

Each step is illustrated with screenshots.

Data Visualization & Insights

Clean data were visualized using the Python pyecharts library and BI tools. Key findings include:

The most expensive and cheapest "big pants" differ mainly in style.

Geographically, Fujian and Zhejiang provinces dominate sales, with Quanzhou (44.28% of Fujian) and Hangzhou (37.02% of Zhejiang) leading.

80% of products are priced below 50 CNY; items above 100 CNY account for less than 2%.

Top‑selling shops are mostly flagship stores, indicating strong brand influence.

Word‑cloud analysis of product names shows common terms such as “short”, “summer”, “male”, and “casual”.

Based on these analyses, the author selected a suitable product and completed the purchase.