Solve Encrypted Python Web Scraping Issues Using Selenium – Step‑by‑Step Guide

Introduction

Hello, I'm PiPi. I recently received a question in a Python community about a web‑scraping issue involving encrypted parameters. I share the problem and solution here.

The page includes encrypted parameters, and the result obtained with a basic crawler was not as expected.

Another screenshot shows the unexpected output.

Solution

A community member suggested using Selenium to handle the dynamic page. The following code demonstrates the approach.

from selenium import webdriver
from selenium.webdriver.common.by import By

url = 'http://ggzyjy.baiyin.gov.cn/InfoPage/TradeInfomation.aspx?state=3'
browser = webdriver.Chrome()
browser.get(url)

li_list = browser.find_elements(By.XPATH, '//ul[@id="tradMainWrap"]/li')
for li in li_list:
    url_detail = li.find_element(By.XPATH, './a').get_attribute('href')
    print(url_detail)

The code successfully retrieved the desired URLs.

Another community member pointed out that the request payload should be a string instead of a dictionary, which fixed the remaining issue.

Conclusion

This article presented a Python web‑scraping problem with encrypted parameters, provided a Selenium‑based solution, and shared practical tips that resolved the issue.