Why Selenium Misses Data and How to Switch to Requests for Reliable Python Web Scraping

The author, a Python enthusiast, shares a recent question from a community about inconsistent data when using Selenium to scrape a website.

Problem

When switching pages with Selenium, sometimes only previously scraped rows appear, leading to missing new data.

Discussion

Participants suggest the issue may stem from using the wrong navigation method (switch window vs get), missing parameters, or flawed pagination logic.

Solution

Instead of Selenium, use the site’s API parameters: request the search result page with the pageNo parameter, extract all pid values via regex, request each gbDetailed page to obtain the PDF download URL ( file_path), construct the final PDF URL, and download the file with requests. Include a conditional check for pages that lack a PDF.

Send a GET request to the search result page with pageNo to paginate.

Use regex to collect all pid identifiers.

Request each gbDetailed page, extract file_path to obtain the PDF file name.

Combine the base URL with the file name to form the PDF URL and download it.

The approach works reliably and avoids the Selenium‑related duplication problem.

Conclusion

The article demonstrates how to troubleshoot a Selenium pagination issue and replace it with a lightweight requests‑based scraper for downloading PDFs.