This tutorial shows how to build a JD.com search URL, fetch the page with urllib, parse the HTML using BeautifulSoup, and reliably extract each product's name, link, image and price while handling missing image URLs and avoiding regex complexity.
This article visually explores seven of Python’s most influential third‑party libraries—NumPy, Pandas, Matplotlib, Requests, Scikit‑learn, Flask, and Beautiful Soup—detailing their core features, typical use cases, and sample code to help developers quickly grasp how each tool can accelerate data processing, visualization, web development, and machine‑learning projects.
This article provides a comprehensive overview of web‑scraping technologies, compares popular tools such as requests, BeautifulSoup and Selenium, introduces AI‑assisted crawling with DeepSeek, and walks through practical steps for using BrightData’s platform to collect industry data, complete with ready‑to‑run Python code.
This tutorial walks you through building a Python web‑scraper for Beike (Lianjia) second‑hand property listings, covering session spoofing, dynamic parameters, pagination, multithreaded detail fetching, data cleaning, and exporting results to Excel in a step‑by‑step manner.
Learn how to efficiently scrape web pages using Python by exploring multiple approaches—including simple requests with BeautifulSoup, fast parsing with lxml, dynamic content extraction with Selenium, and large‑scale crawling with Scrapy—complete with installation steps, code snippets, and detailed explanations.
This article explains why proxy IPs are essential for reliable web crawling, compares dynamic and static residential proxies, and provides step‑by‑step Python code to scrape product titles, prices and links from Snapdeal while demonstrating how to integrate proxies for improved efficiency and security.
This article outlines Python's advantages for web crawling, introduces core libraries such as Requests, BeautifulSoup, and Scrapy, details a step-by-step development workflow, provides practical code examples for extracting news titles, and highlights important considerations and advanced techniques for robust scraper implementation.
Learn why HTML parsing is essential for web scraping, explore three popular Python libraries—BeautifulSoup, lxml, and the built‑in html.parser—covering installation, core usage, advanced techniques, and a comparative analysis to help you choose the right tool for your project.
This tutorial explains how to fetch hot song names and links from NetEase Cloud Music using Python's requests, BeautifulSoup (bs4), and a simple HTML cleanup to overcome malformed tags, providing full code and a working example.
This tutorial explains e‑commerce web scraping fundamentals, covering definitions, tool types, data categories, step‑by‑step Python script creation with Requests, BeautifulSoup, and Selenium, provides sample code for Amazon, Walmart, and eBay, discusses challenges like dynamic pages and anti‑scraping measures, and recommends using specialized scraping APIs.
This article walks you through using Python's requests library together with BeautifulSoup to crawl Baidu search results, extract titles and URLs, and save the data into a CSV file, complete with full source code and sample output images.
This article introduces web crawling fundamentals, demonstrates how to inspect HTML elements, walks through simple examples using urllib, requests, and BeautifulSoup, and culminates in a complete Python script that extracts chapter links and contents from a novel website, saving them to a text file.
This article introduces eleven powerful Python libraries for web scraping—including Requests, BeautifulSoup, Scrapy, Selenium, PyQuery, Lxml, Pandas, Pyppeteer, aiohttp, Faker, and ProxyPool—explains their key features, provides ready‑to‑run code snippets, and demonstrates a real‑world news‑site crawling case study.
This article walks through extracting JSON data from a web page using Python, showcasing multiple code examples with requests, regex, BeautifulSoup, and json handling, and explains how to simplify storage and avoid common parsing pitfalls for reliable web scraping.
This article explains how to build a Python script that searches a comic website, extracts chapter links and image URLs using requests and BeautifulSoup, and downloads the images into organized folders with multithreaded support, while outlining required modules and potential improvements.
This article walks through a five‑minute Python web‑scraping tutorial that fetches recipe pages, parses HTML with urllib and BeautifulSoup, extracts titles and images, and wraps the workflow into a simple console application for random meal selection.
This guide walks you through creating a lightweight Python web scraper that fetches recipe data from a cooking website, covering HTTP requests, HTML parsing with BeautifulSoup, extracting titles and images, and wrapping the process into an interactive console application.
This tutorial explains how to use Python with requests and BeautifulSoup to locate image URLs in a webpage's HTML, extract them via regular expressions, and programmatically download all images to a local folder in an automated batch process.
This tutorial demonstrates how to crawl the DouTuBa emoji website using Python, extract image URLs with regular expressions and BeautifulSoup, and download tens of thousands of images efficiently through a multithreaded downloader.
This tutorial demonstrates how to use Python's requests and BeautifulSoup libraries to query IMDb for movie details such as title, director, cast, rating, and summary, illustrated with a sample lookup for the film Inception.
This article walks through a Python‑based Baidu web‑scraping solution, explaining the problem, showing the debugging process with screenshots, and providing a complete, runnable script that extracts keywords, titles, URLs, descriptions, and site names into a CSV file.
This tutorial demonstrates how to use Python's requests and BeautifulSoup libraries to scrape the titles and release years of the 250 movies listed on Douban, clean the extracted data, output it for Excel, and then create a pivot table and chart to visualize the yearly distribution of top films.
This article walks through a Python web‑crawling issue, explains typical errors like missing data and string comparison pitfalls, and provides a clear BeautifulSoup code example that extracts ranking information, helping readers resolve similar scraping problems.
This article explains how to build a Python web crawler that fetches HTML pages with urllib, parses them using BeautifulSoup to extract image URLs, and downloads the images, covering all three stages with complete code examples and parser options.
This article explains how to use Python's requests and BeautifulSoup libraries to crawl Divnil.com, extract image detail page URLs, parse each page for high‑resolution image links, and download the images into a local folder with a complete, runnable script.
This tutorial walks through building a simple Python web crawler for the QDaily website, covering target analysis, environment setup, SQLite database creation, data extraction with requests and BeautifulSoup, storing articles and comments, performing basic analysis, and visualizing results with a word cloud.
This tutorial explains how to fetch the names and URLs of hot songs from NetEase Cloud Music using Python, addressing XPath limitations by cleaning the HTML and parsing it with BeautifulSoup, and provides a complete, runnable script with example output.
This article explains why XPath fails on NetEase Cloud Music's hot‑song page, presents a regular‑expression solution in Python, provides a complete script, and shows the resulting song names and URLs, while previewing upcoming tutorials using xpath, bs4, and pyquery.
This article demonstrates how to use Python's requests library and BeautifulSoup to crawl Baidu Tieba forums, extracting thread titles and associated images, explains why XPath may fail on irregular HTML, and provides a complete, runnable script with step-by-step implementation details.
This tutorial walks you through using Python and BeautifulSoup to crawl the 8684.cn website, extract city‑specific bus route information, parse classification pages, and collect detailed schedule and station data, while providing reusable code snippets and data‑cleaning suggestions.
This article walks through a Python fan's question about storing JSON from web pages during crawling, presents four step‑by‑step code solutions—including regex, BeautifulSoup, and json.loads—and offers practical tips for clean extraction and reliable scraping.
This article explains how to automate web article extraction by first using BeautifulSoup and urllib to crawl search results and retrieve page links, then demonstrates a much simpler approach with the Python newspaper library to directly download and parse article text, including full installation and code examples.
This article walks through decoding custom font anti‑scraping on a Chinese group‑buying site using Python, covering page fetching, CSS font URL extraction, .woff download, glyph mapping with fontTools, OCR via ddddocr, DOM manipulation with BeautifulSoup, and exporting the cleaned data to Excel.
This tutorial explains how to use Python's requests and BeautifulSoup libraries to fetch a webpage, parse its anchor tags, filter and construct download URLs, and automatically download multiple MP3 files by writing the binary content to local files.
This article demonstrates how to create a Python web crawler that fetches Baidu Baike entries, covering the main program structure, URL manager, page downloader, HTML parser using BeautifulSoup, and output generator, with complete code snippets and sample results.
This article presents a complete Python web crawler example that extracts Baidu Baike entries, detailing the implementation of URL management, page downloading, HTML parsing with BeautifulSoup, data collection, and output generation, along with sample code and usage instructions.
This article explains how to overcome common anti‑scraping defenses such as identity verification and IP rate limiting by spoofing the User‑Agent header and adding request delays, providing complete Python code examples using requests and BeautifulSoup to scrape Douban's Top 250 movies.
This guide walks you through building a Python web‑scraper that extracts NBA player information from the Hupu website, cleans and visualizes the data, and automatically creates Excel files with embedded line charts, covering URL navigation, data parsing with requests and BeautifulSoup, and chart generation with xlsxwriter.
This article walks through creating a Python script that parses web pages, extracts video URLs, and downloads multiple videos efficiently with progress tracking and buffering, providing complete code examples and implementation steps.
This tutorial walks through using Python's urllib and BeautifulSoup libraries to fetch, parse, clean, and export the Fast Track 100 company table into a CSV file, covering installation, page inspection, element extraction, data cleaning, link handling, and file writing.
This article introduces the BeautifulSoup library, explains how to install it, demonstrates core parsing methods such as find, find_all, select, and relationship navigation, and provides a complete example of scraping novel titles from Qidian using Python requests.
This tutorial explains how to analyze the Ant Short‑Term Rental website's anti‑scraping mechanisms, extract the required Cookie and User‑Agent headers, and use Python's urllib2 and BeautifulSoup to reliably crawl rental listings, save the data to CSV, and optionally extend the scraper with Selenium.
In this article we demonstrate how to use Python’s requests and BeautifulSoup to scrape a static webpage listing the world’s top 100 most beautiful women, process the data with pandas and pyecharts for continent, nationality and profession statistics, and apply Baidu’s face‑recognition API to score each celebrity’s attractiveness.
This tutorial demonstrates how to create a Python web scraper using Selenium and BeautifulSoup, covering login automation, HTML retrieval, parsing with html5lib, data extraction from tables, and strategies for handling anti‑scraping measures such as headless browsing and proxy usage.
This tutorial explains how to analyze the Mayi short‑rent website, overcome its anti‑scraping defenses by setting appropriate Cookie and User‑Agent headers, and use Python's urllib2 and BeautifulSoup to extract rental details, store them in CSV, and optionally employ Selenium.
This article explains how to scrape detailed job description and address information from online job portals, use Python libraries such as requests, BeautifulSoup4, and pymongo for crawling, and then clean and normalize the collected data including publish dates, salaries, and work‑experience levels before storing it in MongoDB.
This article demonstrates how to use Python to crawl over 200,000 Bilibili comments, analyze popular memes across different content categories, and presents a step‑by‑step guide to extracting danmu via network requests with code examples for practical data mining.
This tutorial walks through creating a Python script that scrapes video URLs from a web page, downloads them in batches with proper buffering, and displays a real‑time progress bar, covering page analysis, code implementation, and practical tips.
This article provides a step‑by‑step guide on configuring Scrapy, creating a spider project, extracting comic page URLs and images using BeautifulSoup, handling pagination, and saving the downloaded images locally with Python code.
This guide walks you through setting up Python, installing required libraries, capturing Taobao product URLs, logging in, parsing review data with BeautifulSoup, and saving the results, while highlighting best practices to avoid overloading the server.
This article introduces BeautifulSoup, a Python library for parsing HTML/XML, explains how to import it, choose among parsers, demonstrates tag navigation, searching with find/find_all, CSS selection, and tree traversal methods, and provides extensive code examples.
This tutorial walks you through using Python's requests library to fetch a web page and BeautifulSoup4 to parse HTML, covering object creation, common attributes, tag properties, and the find() / find_all() methods for extracting specific content.
This guide introduces the most important Python libraries for web scraping—including requests, urllib3, Selenium, aiohttp, BeautifulSoup, lxml, pyquery, PyMySQL, PyMongo, and redisdump—explaining their core features, typical use cases, and providing concise code examples to help beginners get started quickly.
This guide walks through using Python's requests and BeautifulSoup libraries to simulate Zhihu login, extract user profiles, retrieve answer likers, download avatars, fetch all answers for a question, and store the collected data in a SQLite database.
Learn a step‑by‑step Python solution to locate, download, and store high‑resolution, royalty‑free images from ColorHub by navigating its three‑tier page structure, generating request headers, parsing HTML with BeautifulSoup, and saving files locally, enabling offline PPT creation without copyright concerns.
This tutorial introduces BeautifulSoup, a powerful Python library for parsing HTML and XML, covering installation, basic usage, tag selection, attribute extraction, navigation of parent and sibling nodes, method and CSS selectors, and best‑practice recommendations for efficient web data extraction.
Learn how to install and use Beautiful Soup 4 in Python to parse HTML, navigate the document tree, access tags, attributes, and text, and perform powerful searches with methods like find_all, CSS selectors, and traversal techniques for effective web scraping.
This article introduces Beautiful Soup, a Python library for parsing HTML/XML into a navigable tree, covering installation, object initialization, tag navigation, attribute handling, searching techniques like find_all, CSS selectors, and practical code examples for effective web data extraction.
This article introduces five practical Python libraries—yagmail, requests, psutil, BeautifulSoup, and a collection of useful utils—explaining how each can dramatically reduce code complexity for sending emails, making HTTP calls, monitoring system resources, and parsing HTML.
This article introduces Beautiful Soup, a Python library for parsing HTML/XML into a navigable tree, covering installation, object initialization, tag and attribute access, tree traversal, searching techniques like find_all, find, CSS selectors, and practical code examples.
Learn to automate course enrollment on outdated university portals by simulating login with Python, extracting session data, handling CAPTCHAs, constructing POST requests, and parsing responses, enabling fast, reliable class registration even when schedules conflict.
Learn how to use Python's BeautifulSoup to scrape real‑time weather data from Sohu Weather, format the information, and automatically send it via email using SMTP, with step‑by‑step code examples and tips for handling different email providers.
This guide demonstrates how to use Python's BeautifulSoup library to crawl the Green Breath website, extract real‑time weather and PM2.5 information, handle missing data with conditional checks, and display the results directly in the PyCharm console, providing a practical example of web‑scraping for environmental monitoring.
This article compares four Python web‑scraping selectors—regular expressions, BeautifulSoup, lxml/XPath, and CSS—detailing their strengths, weaknesses, performance, and installation difficulty to help developers choose the most suitable tool for extracting data from sites like JD.com.
Learn step‑by‑step how to use Python’s urllib and BeautifulSoup CSS selectors to fetch JD.com product details—such as name, link, image, and price—by constructing the search URL, parsing the HTML source, and extracting the desired information with concise code examples.
This tutorial shows how to use Python's urllib and BeautifulSoup libraries to encode search keywords, request JD.com pages, parse the HTML tree, and reliably extract product names, links, images, and prices, offering a simpler alternative to complex regular‑expression scrapers.
This guide demonstrates how to combine Python's regex, BeautifulSoup, and Selenium (including Chrome and headless PhantomJS) for powerful web scraping, covering tag matching, handling Ajax, iFrames, cookie management, and practical code examples for extracting and interacting with dynamic web content.
Learn how to scrape static pages, AJAX content, iFrames, and handle cookies using Python libraries such as BeautifulSoup, Selenium, and PhantomJS, while mastering HTTP and URL error handling, CSS‑based element extraction, and practical code examples for robust web data extraction.
This tutorial explains how to collect the names and trading information of every stock listed on the Shanghai and Shenzhen exchanges using Python 3.5, requests, BeautifulSoup and regular expressions, then store the results in a file through a clear three‑step process of fetching the stock list, retrieving each stock's page, parsing the HTML and writing the data.
This tutorial shows iOS developers how to set up Python on macOS, install pip and BeautifulSoup, fetch a web page with urllib2, parse the HTML to locate image tags, extract their URLs, and download the images programmatically for use in an app.
This article walks through building a Python web scraper for Zhihu, covering login simulation, extracting user profiles, answer likers, followers, avatars, and all answers of a question, and storing the collected data in SQLite, while highlighting challenges like captcha and anti‑scraping limits.
