1 views collected around this technical thread.
This article provides a comprehensive overview of web‑scraping technologies, compares popular tools such as requests, BeautifulSoup and Selenium, introduces AI‑assisted crawling with DeepSeek, and walks through practical steps for using BrightData’s platform to collect industry data, complete with ready‑to‑run Python code.
Learn how to efficiently scrape web pages using Python by exploring multiple approaches—including simple requests with BeautifulSoup, fast parsing with lxml, dynamic content extraction with Selenium, and large‑scale crawling with Scrapy—complete with installation steps, code snippets, and detailed explanations.
This article explains various Selenium waiting techniques—including implicit wait, fixed (hard) wait, page load strategies, and JavaScript detection—detailing their principles, advantages, disadvantages, and providing Python code examples to help choose the most suitable approach for stable and efficient automated web testing.
This article introduces ten practical Python-based web scraping methods—including requests, Selenium, Scrapy, Crawley, PySpider, aiohttp, asks, vibora, Pyppeteer, and Fiddler‑plus‑Node reverse engineering—explaining their use cases, advantages, and code examples for efficiently gathering e‑commerce and app data.
This article provides a thorough, step‑by‑step tutorial on using Python Selenium for web automation, covering installation, element locating strategies, form handling, waiting tactics, Page Object design, data‑driven testing, distributed test grids, performance tuning, mobile testing, AI enhancements, best practices, and resources for mastering end‑to‑end testing workflows.
This article introduces beginner-friendly Python web‑scraping methods, covering the simplest urllib/requests approach, adding request headers, inspecting network traffic for hidden data, handling dynamically loaded content with Selenium, and provides links to deeper tutorials for each technique.
This tutorial walks through creating a modular Selenium automation framework with Pytest, covering project structure, utility modules, configuration handling, logging, Page Object Model implementation, test case writing, HTML reporting, and email distribution, all illustrated with complete Python code examples.
This tutorial explains e‑commerce web scraping fundamentals, covering definitions, tool types, data categories, step‑by‑step Python script creation with Requests, BeautifulSoup, and Selenium, provides sample code for Amazon, Walmart, and eBay, discusses challenges like dynamic pages and anti‑scraping measures, and recommends using specialized scraping APIs.
This guide demonstrates how to create a robust online webpage screenshot service using Python, Selenium, and Flask, incorporating security measures like CSRF protection and domain whitelisting, performance enhancements with Redis caching and rate limiting, comprehensive error handling, resource management, and extensibility for future features.
This guide explains how to set up the environment, configure Selenium WebDriver, and run a comprehensive Python script that scrolls through a webpage, captures segmented screenshots, stitches them together with Pillow, and produces a high‑resolution long screenshot, suitable for mobile or web content.
This tutorial demonstrates how to analyze a fund ranking website, construct dynamic URLs, extract six‑digit fund codes, crawl fund holding pages with requests and Selenium, store the results in MongoDB, and finally process the data to identify the most frequently held stocks across thousands of funds.
This article introduces eleven powerful Python libraries for web scraping—including Requests, BeautifulSoup, Scrapy, Selenium, PyQuery, Lxml, Pandas, Pyppeteer, aiohttp, Faker, and ProxyPool—explains their key features, provides ready‑to‑run code snippets, and demonstrates a real‑world news‑site crawling case study.
This article guides functional testers on transitioning to automated testing with Python, covering core concepts, a step‑by‑step learning plan, and ten practical code snippets ranging from unittest and pytest to Selenium web and API automation.
This article explains the ticket‑purchasing workflow, identifies critical timing points for successful ticket grabbing, and presents two common technical approaches—browser automation with Selenium and reverse‑engineering ticket‑booking APIs—to automate the process, while warning about limitations and legal considerations.
This tutorial demonstrates how to install Selenium, log into QQ Space, and programmatically retrieve both status updates and album photos using Python's Selenium and BeautifulSoup libraries, including detailed code examples for login, scrolling, and image downloading.
This article introduces ten practical Python techniques—including requests, Selenium, Scrapy, Crawley, PySpider, aiohttp, asks, vibora, Pyppeteer, and Fiddler‑based reverse engineering—to efficiently collect e‑commerce and app data while addressing common challenges such as IP blocking, captchas, and authentication.
The article introduces the Python mechanize library, explains its capabilities for automating web interactions such as form submission, link navigation, cookie management, and redirects, provides basic usage examples, detailed sample code for various testing scenarios, and notes its limitations compared to Selenium.
This article provides a step‑by‑step tutorial on setting up Python and Selenium, installing the appropriate browser driver, and writing a script to automate ticket purchasing on a web platform, including code examples, execution flow, and practical tips for handling different site structures.
This article explains how to perform UI automation testing across various computer screen resolutions using Python, covering compatible frameworks like Selenium and Appium, writing resolution‑agnostic scripts, creating resolution matrices, dynamic resizing, visual verification, and integrating tests into CI pipelines.
This article introduces seven practical Python tools—including Pandas, Selenium, Flask, Scrapy, Requests, Faker, and Pillow—explaining their core features, typical use cases, and providing ready‑to‑run code snippets to help developers automate tasks and accelerate project development.