This tutorial walks through installing Scrapy on Windows, resolving frequent errors like missing lxml by using unofficial Windows binaries, fixing Microsoft Visual C++ 14.0 requirement via Build Tools, and a workaround for unsupported Twisted wheels by renaming .whl to .zip and extracting.
This article reviews the four main Python selectors used for web scraping—regular expressions, BeautifulSoup, lxml/XPath, and CSS—detailing their installation, parsing behavior, advantages, drawbacks, and a performance comparison table to help developers choose the most suitable tool for their projects.
This article walks through a Python web‑scraping problem where an incorrect XPath selector caused unexpected results, demonstrates the faulty code, explains why the XPath was wrong, and provides a corrected script with proper headers to reliably extract the desired text.
This article walks through a real‑world Python web‑scraping problem, shows why the original XPath selector fails, provides corrected code with a working XPath expression, and highlights best practices such as adding request headers for reliable crawling.
This article walks through a Python web‑scraping problem, demonstrates why the original XPath extraction returns noisy or missing price data, and provides multiple refined code solutions—including filtering empty entries, correcting XPath selectors, and using map‑filter techniques—to produce clean, formatted price lists.
Learn why HTML parsing is essential for web scraping, explore three popular Python libraries—BeautifulSoup, lxml, and the built‑in html.parser—covering installation, core usage, advanced techniques, and a comparative analysis to help you choose the right tool for your project.
This tutorial shows how to use Python to scrape a daily article from a website, format the content, and automatically send it via email using SMTP, with full source code and guidance for scheduling the script.
This article explains why XPath fails on NetEase Cloud Music pages, demonstrates a regex‑based Python solution with full source code, shows sample output, and outlines future tutorials using XPath, BeautifulSoup, and PyQuery to strengthen Python selector skills.
This tutorial walks you through creating a Python script that crawls an online dictionary to fetch word‑meaning pairs, formats them, and automatically emails the list each day, using libraries like lxml, requests, BeautifulSoup, and smtplib for scheduling and delivery.
Learn how to scrape Baidu search result titles and URLs using Python's requests library and lxml's xpath parsing, complete with a ready-to-run script, CSV output, and step-by-step explanation that builds on earlier regex and BeautifulSoup methods.
This article walks through a typical Python web‑scraping issue caused by incorrect URL concatenation and XPath mismatches, explains the debugging process, and provides complete, ready‑to‑run code along with a practical urljoin example to prevent similar errors.
This article walks you through using Python's requests and lxml libraries to crawl a fund website, extract stock IDs, fetch detailed fund information, process the data, and save it to CSV for further analysis, complete with code snippets and screenshots.
This article provides a comprehensive Python tutorial for scraping second‑hand housing community data from Anjuke, covering city selection, URL collection, HTML parsing with lxml, data cleaning, CSV export, and full‑city crawling strategies, complete with runnable code examples.
This article walks through a Python web‑scraping solution for downloading individual files from a site, presenting the problem, multiple approaches, and a complete, ready‑to‑run script, while also offering tips for handling large data and avoiding IP blocks.
This article demonstrates how to use Python's requests, lxml, and pandas libraries to crawl GDP data from a website, parse the HTML tables, and efficiently write the extracted rankings, regions, GDP values, and years into a CSV file, providing a complete, runnable example for web scraping beginners.
This article walks through a Python web‑scraping solution for extracting and downloading audio files from a WeChat page, explains the problem, shows the implementation steps with screenshots, and provides the complete, ready‑to‑run code.
This article walks through a Python web‑scraping solution that extracts GDP data from a website using requests, lxml, and pandas, then stores the results in a CSV file, providing complete code and step‑by‑step explanation for readers.
This article walks through a Python web‑scraping solution that fetches GDP data from a website using the requests library, parses HTML with lxml, and demonstrates two approaches—manual XPath extraction and a streamlined pandas.read_html method—while providing complete code snippets and tips for handling pagination and data storage.
This article walks through diagnosing and fixing an XPath extraction issue in a Python web scraper, showing the original faulty code, the corrected selector, and the final working script with sample outputs, while emphasizing best practices like setting request headers.
An experienced Python developer walks through a real-world web‑scraping issue, showing how a faulty XPath selector caused empty results, then provides corrected code, execution screenshots, and best practices like adding request headers, helping readers quickly resolve similar problems.
This article walks through a Python web‑scraping problem where price data extracted via XPath contains empty lines and unwanted characters, presenting multiple code solutions—including list comprehensions, map‑filter, and conditional printing—to clean and correctly display the product prices.
This article demonstrates how to use Python's requests and lxml libraries to scrape web pages and apply regular expressions for extracting both integer and floating‑point numbers, providing clear code examples and step‑by‑step explanations.
This tutorial shows how to create a Python web scraper that fetches a daily article, formats it, and sends it via email using SMTP, including the full source code and step‑by‑step implementation details.
Learn how to extract the names and URLs of popular NetEase Music tracks using Python's requests, lxml's XPath, and a simple HTML cleanup technique, with full code examples and tips for handling non‑standard HTML responses.
This tutorial shows how to create a Python web crawler that fetches English words, formats them, and automatically emails the list each day using SMTP and a scheduling library, providing a practical solution for daily vocabulary practice.
Learn how to scrape Baidu Tieba thread titles and associated images using Python's requests library and XPath expressions, with a complete example script that handles malformed HTML, removes comment markers, and outputs title‑image pairs for further processing.
This tutorial demonstrates a complete Python script that uses requests and lxml's XPath to fetch Baidu search result titles and URLs, saves them to a CSV file, and includes sample output screenshots for verification.
This article walks through a real-world Python web‑scraping issue where an extra slash in the URL caused failures, demonstrates debugging with try/except, shows a complete recipe‑crawling script using requests, lxml, and UserAgent, and explains proper URL joining with urllib.parse.urljoin.
Learn how to use Python's requests and lxml libraries to scrape fund codes and detailed financial data from a public fund website, extract the information with simple regex and XPath, and store the results in a CSV file for further analysis.
This tutorial walks you through creating a Python web scraper that logs into Weibo, extracts the real‑time hot search list and top comments using requests, lxml and pandas, saves the data to Excel, and schedules the script to run hourly with Windows Task Scheduler.
This tutorial explains the fundamental concepts and parsing principles of XPath, shows how to set up the Python lxml environment, demonstrates instantiating etree objects, details XPath expression syntax, and provides multiple real‑world web‑scraping examples with complete code snippets.
This tutorial shows how to use Python's requests, lxml, and pandas libraries to crawl Weibo's real‑time hot search list, extract ranking, popularity, titles, and detailed comments, and schedule the script to run automatically each hour.
This tutorial shows how to crawl the China Meteorological Administration website to obtain today's weather and a six‑day forecast, handle Chinese encoding, extract data with XPath, and save the results into a CSV file using Python's requests and lxml libraries.
This tutorial demonstrates how to use Python's requests and lxml libraries to scrape quotes, authors, and tags from quotes.toscrape.com, parse the HTML with XPath, and save the extracted data into a CSV file.
The guide explains how to use Python libraries such as requests, pandas, lxml, and regex to automatically paginate through a website, extract PDF file names and URLs, create organized folders, and download all files in bulk with minimal manual effort.
This article demonstrates how to build a Python web scraper using requests, lxml, regular expressions, and tqdm to locate, extract, and download video files from a VIP anime website, covering header configuration, XPath parsing, URL reconstruction, and file saving.
This tutorial demonstrates how to use Python's requests and lxml libraries to scrape static web pages, extract image URLs via XPath, convert thumbnail links to full‑size URLs, and download the images to a local folder, providing a complete end‑to‑end example.
This article provides a comprehensive tutorial on XPath concepts, node types, syntax, axes, predicates, and operators, followed by a detailed introduction to the fast Python lxml library, its installation, usage for both offline and online HTML parsing, and practical code examples for extracting elements and attributes.
This article explains how to use Python's requests and lxml libraries to scrape NetEase Cloud Music's song list, extract song IDs, construct download URLs, and save the audio files locally, while also discussing common obstacles such as anti‑scraping measures and network errors.
This article walks through using Python's requests and lxml libraries to fetch, parse, and download songs from NetEase Cloud Music, detailing the implementation steps, code snippets, encountered obstacles, and tips for handling anti‑scraping measures.
This article walks you through building a Python web‑scraper that queries Baidu for the keyword “grain,” extracts article titles, abstracts and links using Requests and lxml, saves the results to a text file, and shows the execution output with screenshots.
This article demonstrates how to use Python's requests, fake_useragent, and lxml libraries to batch‑scrape nearly a thousand new‑house listings from the 惠民之家 website, extracting 41 fields such as name, price, layout, opening date, plot ratio and green ratio, while handling pagination and anti‑scraping measures.
This article introduces the fundamentals of XPath syntax, common rules, and example expressions, then explains how to use the lxml library in Python for HTML/XML parsing, including practical tips and a complete code example for extracting links and text from a sample document.
This tutorial explains how to use Python's requests and lxml libraries to crawl a website, bypass SSL verification, extract audio file URLs, and download sound effect MP3s in bulk, providing complete code snippets and practical tips for safe web scraping.
This tutorial walks you through using Python's requests, lxml, and JSON parsing to automatically crawl the King of Glory website, extract hero skin URLs, and download the images into organized folders while displaying progress in the console.
This tutorial explains how to crawl 58.com job listings with Python, extract location, company, and salary information, handle anti‑scraping defenses using realistic headers and random User‑Agents, and save the results into a text file.
This tutorial explains how to use Python's requests, lxml, and fake_useragent libraries to crawl the free novel website "无限小说网", extract each novel's download link, and automatically download the corresponding text files.
This tutorial walks you through building a Python web scraper that extracts recipe names, ingredients, and download links from the XiaChuFang cooking website, handling anti‑scraping measures with custom headers and fake user agents, and saves the collected data into a Word document for future use.
This tutorial walks you through using Python's requests, lxml, and fake_useragent libraries to crawl Qyer travel city pages, extract city names, images, and hotspots, and store the results in a CSV file while handling pagination and request headers.
This guide walks you through building a Python web‑scraping project that overcomes common anti‑scraping defenses, extracts QQ numbers from a target site, and saves the results into structured Excel files, while highlighting required libraries, request headers, and AJAX handling techniques.
This tutorial walks you through creating a Python web scraper that fetches image URLs from 51miz.com using requests and lxml, filters them with regular expressions, downloads the images, and demonstrates the complete workflow with code snippets and screenshots.
This tutorial demonstrates how to use Python's requests, lxml, and pprint libraries to crawl the Hao123 travel website, extract attraction names, opening times, reviews, and prices, and display the results, while providing complete code examples and practical tips.
This tutorial walks through building a Python web scraper that fetches anime torrent links from the Mikan Project site, handles pagination, bypasses anti‑scraping measures, parses pages with lxml, and saves the .torrent files locally, complete with code snippets and screenshots.
This tutorial walks you through using Python's requests and lxml libraries to reverse‑engineer the Youdao mobile translation interface, construct the required form parameters, send POST requests, parse the returned HTML with XPath, and display translated results for multiple languages.
This tutorial explains how to build a Python web‑scraper that searches Baidu Tieba by keyword, bypasses anti‑crawling measures, extracts image and video URLs with XPath, and saves the media files locally, complete with code examples and setup instructions.
This tutorial explains how to use Python's requests and lxml libraries to crawl a public image site, parse pagination URLs, retrieve high‑resolution 王者荣耀 pictures, and save them locally in batches, while highlighting key implementation steps and best‑practice cautions.
This tutorial walks you through extracting image URLs from a webpage using XPath, constructing full URLs, and automating batch downloads with Python's requests, lxml, and fake_useragent libraries, including code snippets and practical tips for handling files and headers.
This tutorial explains how to automate bulk image downloads from a website using Python's requests, lxml, and fake_useragent libraries, covering environment setup, header configuration, URL formatting, page iteration, HTML parsing, and download logic in a clear, practical example.
This comprehensive guide walks you through Python web crawling fundamentals, covering request handling, URL encoding, regular expressions, the requests library, XPath parsing, and lxml, complete with code snippets and practical examples to help you build effective scrapers.
This guide introduces the most important Python libraries for web scraping—including requests, urllib3, Selenium, aiohttp, BeautifulSoup, lxml, pyquery, PyMySQL, PyMongo, and redisdump—explaining their core features, typical use cases, and providing concise code examples to help beginners get started quickly.
This tutorial introduces BeautifulSoup, a powerful Python library for parsing HTML and XML, covering installation, basic usage, tag selection, attribute extraction, navigation of parent and sibling nodes, method and CSS selectors, and best‑practice recommendations for efficient web data extraction.
This article compares four Python web‑scraping selectors—regular expressions, BeautifulSoup, lxml/XPath, and CSS—detailing their strengths, weaknesses, performance, and installation difficulty to help developers choose the most suitable tool for extracting data from sites like JD.com.
This guide explains how to use XPath efficiently for web scraping, covering node selection, axes, functions, numeric comparisons, and advanced combinations, while emphasizing concise and readable expressions to improve performance and maintainability.
