1 views collected around this technical thread.
This article provides a comprehensive overview of web‑scraping technologies, compares popular tools such as requests, BeautifulSoup and Selenium, introduces AI‑assisted crawling with DeepSeek, and walks through practical steps for using BrightData’s platform to collect industry data, complete with ready‑to‑run Python code.
This tutorial walks you through building a Python web‑scraper for Beike (Lianjia) second‑hand property listings, covering session spoofing, dynamic parameters, pagination, multithreaded detail fetching, data cleaning, and exporting results to Excel in a step‑by‑step manner.
Learn how to efficiently scrape web pages using Python by exploring multiple approaches—including simple requests with BeautifulSoup, fast parsing with lxml, dynamic content extraction with Selenium, and large‑scale crawling with Scrapy—complete with installation steps, code snippets, and detailed explanations.
This article explains why proxy IPs are essential for reliable web crawling, compares dynamic and static residential proxies, and provides step‑by‑step Python code to scrape product titles, prices and links from Snapdeal while demonstrating how to integrate proxies for improved efficiency and security.
This article outlines Python's advantages for web crawling, introduces core libraries such as Requests, BeautifulSoup, and Scrapy, details a step-by-step development workflow, provides practical code examples for extracting news titles, and highlights important considerations and advanced techniques for robust scraper implementation.
Learn why HTML parsing is essential for web scraping, explore three popular Python libraries—BeautifulSoup, lxml, and the built‑in html.parser—covering installation, core usage, advanced techniques, and a comparative analysis to help you choose the right tool for your project.
This article explains how to use Python's requests, BeautifulSoup, and Pillow libraries to crawl high‑resolution anime pictures from the Huashi6 website, extract their webp URLs via regex, download them, and convert the images to the more common png format for personal use.
This tutorial explains e‑commerce web scraping fundamentals, covering definitions, tool types, data categories, step‑by‑step Python script creation with Requests, BeautifulSoup, and Selenium, provides sample code for Amazon, Walmart, and eBay, discusses challenges like dynamic pages and anti‑scraping measures, and recommends using specialized scraping APIs.
This article introduces web crawling fundamentals, demonstrates how to inspect HTML elements, walks through simple examples using urllib, requests, and BeautifulSoup, and culminates in a complete Python script that extracts chapter links and contents from a novel website, saving them to a text file.
This article introduces eleven powerful Python libraries for web scraping—including Requests, BeautifulSoup, Scrapy, Selenium, PyQuery, Lxml, Pandas, Pyppeteer, aiohttp, Faker, and ProxyPool—explains their key features, provides ready‑to‑run code snippets, and demonstrates a real‑world news‑site crawling case study.
This article explains how to build a Python script that searches a comic website, extracts chapter links and image URLs using requests and BeautifulSoup, and downloads the images into organized folders with multithreaded support, while outlining required modules and potential improvements.
This article walks through a five‑minute Python web‑scraping tutorial that fetches recipe pages, parses HTML with urllib and BeautifulSoup, extracts titles and images, and wraps the workflow into a simple console application for random meal selection.
This article explains how to automate Google SEO keyword rank checking by writing a Python script that sends HTTP requests, parses the search results with BeautifulSoup, iterates through results to find a target domain, and handles errors, providing a free alternative to costly SEO tools.
This tutorial explains how to use Python with requests and BeautifulSoup to locate image URLs in a webpage's HTML, extract them via regular expressions, and programmatically download all images to a local folder in an automated batch process.
This tutorial demonstrates how to crawl the DouTuBa emoji website using Python, extract image URLs with regular expressions and BeautifulSoup, and download tens of thousands of images efficiently through a multithreaded downloader.
This tutorial demonstrates how to use Python's requests and BeautifulSoup libraries to query IMDb for movie details such as title, director, cast, rating, and summary, illustrated with a sample lookup for the film Inception.
This tutorial demonstrates how to use Python's requests and BeautifulSoup libraries to scrape the titles and release years of the 250 movies listed on Douban, clean the extracted data, output it for Excel, and then create a pivot table and chart to visualize the yearly distribution of top films.
This article explains how to use Python's requests and BeautifulSoup libraries to crawl Divnil.com, extract image detail page URLs, parse each page for high‑resolution image links, and download the images into a local folder with a complete, runnable script.
This tutorial explains the fundamentals of web crawling by distinguishing static and dynamic pages, outlining a four‑step process for scraping static sites, and providing a complete Python example that extracts historical weather data, parses HTML with BeautifulSoup, and stores results in CSV files.
This article demonstrates how to use Python's requests library and BeautifulSoup to inspect webpage source, set request headers, fetch weather page HTML, parse it with CSS selectors, extract daytime and nighttime temperatures, and extend the script to handle multiple cities, providing complete code examples.