This article walks through a real-world Python web‑scraping issue where an extra slash in the URL caused failures, demonstrates debugging with try/except, shows a complete recipe‑crawling script using requests, lxml, and UserAgent, and explains proper URL joining with urllib.parse.urljoin.
Learn how to use Python's requests and lxml libraries to scrape fund codes and detailed financial data from a public fund website, extract the information with simple regex and XPath, and store the results in a CSV file for further analysis.
This article introduces the Python Requests library, explains how to install it, demonstrates basic HTTP methods such as GET and POST with code examples, and explores advanced use‑cases like handling JSON, binary data, custom headers, cookies, sessions, SSL verification, proxies, timeouts, and authentication.
This tutorial walks you through creating a Python web scraper that logs into Weibo, extracts the real‑time hot search list and top comments using requests, lxml and pandas, saves the data to Excel, and schedules the script to run hourly with Windows Task Scheduler.
This article walks through a Python fan's question about storing JSON from web pages during crawling, presents four step‑by‑step code solutions—including regex, BeautifulSoup, and json.loads—and offers practical tips for clean extraction and reliable scraping.
This tutorial walks through creating a Python‑based local QR‑code login scraper for QQ Music, covering the extraction of dynamic parameters, handling of encrypted cookies, displaying and removing QR images, and ultimately obtaining a usable session for further automation.
This article explains how to identify gzip‑compressed responses that may be gzip bombs, how to inspect HTTP headers and raw response data using Python's requests library, and provides command‑line and code examples for measuring compressed and uncompressed sizes without triggering decompression.
This guide explains how to use Python's requests library to brute‑force DVWA login pages at low and medium security levels, detailing code snippets, request headers, payload construction, response handling, and result logging to CSV or TXT files.
This tutorial demonstrates how to programmatically obtain a QQ Music QR‑code login session using Python by fetching the QR image, extracting and decrypting the required cookies, handling dynamic parameters, and completing the login flow with detailed code examples and explanations.
This tutorial shows how to crawl the China Meteorological Administration website to obtain today's weather and a six‑day forecast, handle Chinese encoding, extract data with XPath, and save the results into a CSV file using Python's requests and lxml libraries.
This tutorial demonstrates how to use Python's requests library to programmatically log into Weibo, publish a status update, and automatically like posts by handling cookies, sessions, and the required st token, complete with full code examples.
This article explains why Chinese characters often appear as garbled text when crawling websites with Python and provides three practical solutions—using response.content, manually setting the response encoding, and applying a generic encode‑decode trick—to reliably decode the data.
This article explains why the Python requests library may install correctly yet fail to import, covering virtual‑environment issues, filename conflicts, and attribute errors, and provides step‑by‑step solutions to get the import working again.
This article walks through a common Python requests encoding error, explains why it occurs due to mismatched UTF-8 and GBK settings, and provides step‑by‑step instructions—including PyCharm configuration and code snippets—to resolve the issue permanently.
This tutorial demonstrates how to use Python's requests and lxml libraries to scrape static web pages, extract image URLs via XPath, convert thumbnail links to full‑size URLs, and download the images to a local folder, providing a complete end‑to‑end example.
This article explains how to create a Python web crawler that automatically generates URLs for a drama‑download site, filters out invalid pages, extracts ed2k links using requests and regular expressions, saves them to text files, and employs multithreading to speed up processing, while discussing challenges such as duplicate URLs and filename sanitization.
This article describes how to build a Python web crawler that automatically generates numeric URLs, checks their validity, extracts download links for TV dramas from cn163.net, saves them to text files, and discusses practical challenges such as regex parsing, filename restrictions, and multithreading performance.
This article provides a Python tutorial that explains how to generate Eastmoney‑specific security IDs, request minute‑level K‑line data for a given stock code via the Eastmoney API, process the response into a pandas DataFrame, and continuously fetch and save data until market close.
This article introduces the Python filestools library, demonstrating how to install it and use its four core utilities—tree‑style directory listing, text file diff comparison, image watermarking, and converting cURL commands to requests code—complete with command‑line and Jupyter examples.
This tutorial explains how to install the Python requests library and use it to perform GET and POST HTTP requests, detailing required parameters, headers, and providing complete code examples for calling remote APIs and handling responses.
This article provides a comprehensive Python tutorial covering how to send GET and POST requests, handle HTTPS verification, perform file uploads and downloads, set request timeouts, use basic and token authentication, manage sessions, generate signatures, and organize a test automation framework with configuration, common utilities, test data, test cases, reporting, and execution scripts.
This tutorial walks you through using Python's requests library to inspect Kugou Music's web requests, simulate them, extract MP3 URLs, and download songs, covering preparation, goal setting, implementation steps, and a concise project summary.
This tutorial walks you through analyzing a JavaScript‑driven website, discovering the hidden API, configuring request headers with Postman, and writing a Python scraper that extracts book titles, authors, codes, and prices from epubit.com.
This article explains how to overcome common anti‑scraping defenses such as identity verification and IP rate limiting by spoofing the User‑Agent header and adding request delays, providing complete Python code examples using requests and BeautifulSoup to scrape Douban's Top 250 movies.
This guide walks you through analyzing Toutiao’s Ajax requests, extracting image URLs, and using Python’s requests library to download street‑photo pictures in bulk, complete with code snippets and tips for pagination and future enhancements.
This tutorial walks you through creating an interactive Excel weather dashboard using Python's xlwings and requests libraries, covering API selection, data retrieval, code integration, Excel macro setup, UI design, and dynamic updates for multiple Chinese cities.
This article introduces seven powerful Python tools—including Pandas, Selenium, Flask, Scrapy, Requests, Faker, and Pillow—explaining their core features and providing ready-to-use code snippets to help developers automate data analysis, testing, web development, crawling, API calls, fake data generation, and image processing.
This tutorial explains how to download files, webpages, and Amazon S3 objects using Python, covering the requests and wget modules, handling redirects, chunked and parallel downloads, progress bars, proxy support, urllib/urllib3 usage, and asynchronous downloading with asyncio.
This guide demonstrates how to use Python's requests library for sending GET, POST, and HTTPS requests, handling file uploads and downloads, managing timeouts, performing authentication, session and token handling, generating signatures, and organizing a complete API test automation framework.
This article introduces the BeautifulSoup library, explains how to install it, demonstrates core parsing methods such as find, find_all, select, and relationship navigation, and provides a complete example of scraping novel titles from Qidian using Python requests.
This article explains Python's context manager mechanism, covering the basic with statement, custom __enter__/__exit__ classes, the contextlib.contextmanager decorator, nesting, combining multiple managers with ExitStack, and practical applications such as SQLAlchemy session handling, exception management, and persistent HTTP requests.
This article details the step‑by‑step creation of a Python‑based API automation testing framework, covering project structure, configuration files, a reusable logging module, HTTP request handling with requests, common utilities for Excel and XML, MySQL database access, email reporting, and the test runner setup.
This article explains how to use Python's requests and lxml libraries to scrape NetEase Cloud Music's song list, extract song IDs, construct download URLs, and save the audio files locally, while also discussing common obstacles such as anti‑scraping measures and network errors.
This guide introduces the Python requests library, showing how to import it, perform GET and POST requests, handle query parameters, form data, response content, encoding, proxies, status codes, custom headers, and session objects with clear code examples.
This article walks through using Python's requests and lxml libraries to fetch, parse, and download songs from NetEase Cloud Music, detailing the implementation steps, code snippets, encountered obstacles, and tips for handling anti‑scraping measures.
This tutorial demonstrates how to use Python's requests and BeautifulSoup libraries to scrape the titles and release years of the top 250 movies from Douban, clean the data, export it to Excel, and then create pivot tables and charts to visualize yearly film production trends.
This article walks you through building a Python web‑scraper that queries Baidu for the keyword “grain,” extracts article titles, abstracts and links using Requests and lxml, saves the results to a text file, and shows the execution output with screenshots.
This tutorial shows how to install the Python requests library and use it to perform GET and POST HTTP requests, covering required parameters, headers, and example code for handling responses.
This guide shows how to install the Python requests library and use it to perform GET and POST HTTP requests, explaining required parameters, headers, and providing complete code examples for calling remote APIs.
This article demonstrates how to use Python's requests, fake_useragent, and lxml libraries to batch‑scrape nearly a thousand new‑house listings from the 惠民之家 website, extracting 41 fields such as name, price, layout, opening date, plot ratio and green ratio, while handling pagination and anti‑scraping measures.
This tutorial explains what an API is, shows how to install and use the Python requests library to make GET requests, interprets common HTTP status codes, and demonstrates parsing JSON data from the Open Notify ISS API using the built‑in json module.
This tutorial explains how to use Python's requests, lxml, and urllib libraries to search Baidu Tieba by keyword, bypass anti‑crawling measures, extract image and video URLs with XPath, and save the media files locally, complete with code examples and screenshots.
This article explains how to use Python's requests library and CSV handling to automate API testing of payment interfaces, focusing on correctly processing special characters in usernames and account names to ensure secure and fast transactions.
This guide walks you through setting up a Python environment, installing required libraries, constructing a FilmSky scraper class, handling pagination, parsing HTML with regex, and saving movie titles and download links, enabling you to browse and download movies from the FilmSky website efficiently.
This guide walks you through using the Python httpx library to perform various HTTP methods—including GET, POST, PUT, DELETE, HEAD, and OPTIONS—handle query parameters, decode responses, work with JSON, custom headers, form data, file uploads, streaming, cookies, redirects, and authentication, all with clear code examples.
This guide walks through analyzing a Tenda router's web interface to discover its data‑fetching URLs, extracting all available modules, and then using Python's requests library to retrieve the information and expose it through a simple Flask API, with both object‑oriented and functional code examples.
This tutorial walks you through using various Python modules—including requests, wget, urllib, urllib3, boto3, and asyncio—to download regular files, web pages, Amazon S3 objects, handle redirects, large files, parallel downloads, proxies, and progress bars, all with clear code examples.
This tutorial walks you through using Python's requests, lxml, and JSON parsing to automatically crawl the King of Glory website, extract hero skin URLs, and download the images into organized folders while displaying progress in the console.
This article demonstrates how to use Python to crawl over 200,000 Bilibili comments, analyze popular memes across different content categories, and presents a step‑by‑step guide to extracting danmu via network requests with code examples for practical data mining.
This tutorial introduces Python's Requests library, covering installation, basic GET and POST requests, adding headers and parameters, handling proxies, cookies, sessions, and disabling SSL verification, with clear code examples for each feature.
This tutorial walks you through creating a Python web scraper that extracts images from a target site, handles anti‑scraping measures, saves them into categorized folders, and logs the download results, while explaining the required libraries, code structure, and best practices.
Learn how to build a Python web scraper that fetches upcoming movie details from Maoyan.com, covering environment setup, URL pagination, random User‑Agent handling, HTML parsing with XPath, data extraction, and result display, while highlighting best practices and limitations.
This tutorial walks through building a simple Python web crawler that searches the Pansou site, handles AJAX‑loaded JSON data, extracts file links, and enables interactive downloading, illustrating core backend techniques such as HTTP GET requests, JSON parsing, and user‑driven pagination.
This tutorial explains how to analyze Meituan food page URLs, use browser developer tools to locate AJAX JSON responses, construct Python requests with proper headers, extract restaurant information via regular expressions, and save the results to a local file.
This tutorial explains how to crawl 58.com job listings with Python, extract location, company, and salary information, handle anti‑scraping defenses using realistic headers and random User‑Agents, and save the results into a text file.
This tutorial introduces Python’s Requests library, covering installation, core methods like GET, POST, PUT, PATCH, DELETE, detailed parameters, session handling, exception management, header customization, proxy usage, and practical code examples to empower effective web scraping.
This tutorial walks through creating a Python script that scrapes video URLs from a web page, downloads them in batches with proper buffering, and displays a real‑time progress bar, covering page analysis, code implementation, and practical tips.
This tutorial walks you through building a Python web scraper that extracts recipe names, ingredients, and download links from the XiaChuFang cooking website, handling anti‑scraping measures with custom headers and fake user agents, and saves the collected data into a Word document for future use.
This tutorial walks you through creating a Python web scraper that fetches image URLs from 51miz.com using requests and lxml, filters them with regular expressions, downloads the images, and demonstrates the complete workflow with code snippets and screenshots.
This tutorial explains how to create a Python CLI application named tickets that parses command-line arguments, retrieves station codes, queries the 12306 railway API for ticket data, and displays the results in a formatted table using libraries such as docopt, requests, and prettytable.
This tutorial demonstrates how to use Python's requests, threading, and queue modules to build a multithreaded crawler that extracts game names, download links, and execution time from the Xiaomi App Store, complete with code examples and performance tips.
This tutorial explains how to build a Python web‑scraper that searches Baidu Tieba by keyword, bypasses anti‑crawling measures, extracts image and video URLs with XPath, and saves the media files locally, complete with code examples and setup instructions.
This tutorial explains how to use Python's requests and lxml libraries to crawl a public image site, parse pagination URLs, retrieve high‑resolution 王者荣耀 pictures, and save them locally in batches, while highlighting key implementation steps and best‑practice cautions.
This guide walks you through setting up Python, installing required libraries, capturing Taobao product URLs, logging in, parsing review data with BeautifulSoup, and saving the results, while highlighting best practices to avoid overloading the server.
This guide walks you through setting up a Python environment, installing required libraries, writing a FilmSky class with request handling, parsing HTML using regular expressions, and saving movie titles and download links, providing a practical example of web crawling for movie sites.
This tutorial walks you through extracting image URLs from a webpage using XPath, constructing full URLs, and automating batch downloads with Python's requests, lxml, and fake_useragent libraries, including code snippets and practical tips for handling files and headers.
This tutorial explains how to automate bulk image downloads from a website using Python's requests, lxml, and fake_useragent libraries, covering environment setup, header configuration, URL formatting, page iteration, HTML parsing, and download logic in a clear, practical example.
This tutorial walks through multiple Python approaches for downloading files—including simple requests and wget calls, handling redirects, large and multi‑file downloads, proxy usage, urllib/urllib3 methods, and asynchronous techniques—providing complete code snippets and practical tips for each scenario.
This article explores the internal call chain of the Python Requests library, explains how the timeout argument is propagated through urllib3’s pool manager and HTTPConnection objects, and shows the relevant source code that makes socket-level timeouts work for HTTP requests.
This tutorial explains how to obtain free proxy IPs, extract their addresses using Python's requests and BeautifulSoup, and continuously validate them to overcome anti‑scraping restrictions when crawling sites such as Baidu Baike for data mining tasks.
This comprehensive guide walks you through Python web crawling fundamentals, covering request handling, URL encoding, regular expressions, the requests library, XPath parsing, and lxml, complete with code snippets and practical examples to help you build effective scrapers.
This guide walks you through the fundamentals of building a web crawler in Python, covering how to fetch pages with the Requests library, extract data using regular expressions and XPath, and provides practical code examples for each step.
This guide explains the fundamentals of web crawling, covering crawler types, the Robots.txt protocol, Python's urllib and urllib3 modules, the requests library, handling HTTP methods, user‑agents, HTTPS certificates, and practical code examples for extracting data from websites.
This article demonstrates how to automate e‑commerce test data creation by retrieving schema information with SQLAlchemy ORM, defining mapping classes, generating realistic data using Faker, and submitting it via HTTP requests with the Requests library, covering database interactions, data assembly, and API communication.
This guide introduces the most important Python libraries for web scraping—including requests, urllib3, Selenium, aiohttp, BeautifulSoup, lxml, pyquery, PyMySQL, PyMongo, and redisdump—explaining their core features, typical use cases, and providing concise code examples to help beginners get started quickly.
This guide demonstrates how to quickly build Python-based HTTP/HTTPS API automation test scripts using requests for sending requests, pandas for handling test case data, and openpyxl for exporting results to Excel, covering request construction, response handling, data comparison, and result styling.
Learn how to start web scraping with Python by mastering the three core steps—fetching, analyzing, and storing data—using urllib and requests, handling login, evading anti‑scraping measures like user‑agents and IP proxies, and saving results to JSON, CSV, or MongoDB.
This tutorial walks you through analyzing JD product pages, extracting comment data via requests with proper headers, handling pagination, saving results, cleaning text using jieba, and visualizing frequent terms as a word cloud, all illustrated with step‑by‑step screenshots and code snippets.
This article describes how the author built a Python web crawler to automatically generate numeric URLs, fetch TV drama pages from the 天天美剧 site, extract ed2k download links using regular expressions, and save them into organized text files, streamlining the download process with Thunder.
This article describes a real‑time monitoring solution for large numbers of H5 pages that combines Python's Requests library for data crawling with a headless Chrome browser driven by Puppeteer to detect resource errors, API failures, and DOM anomalies, automatically alerting stakeholders.
This article introduces five practical Python libraries—yagmail, requests, psutil, BeautifulSoup, and a collection of useful utils—explaining how each can dramatically reduce code complexity for sending emails, making HTTP calls, monitoring system resources, and parsing HTML.
This article explains how to create a Python web‑scraper that automatically generates URLs, fetches drama pages from a download site, extracts ed2k links with regular expressions, saves them to text files, and handles missing pages and filename restrictions efficiently.
This tutorial introduces web crawlers, explains how to inspect and temporarily modify page HTML using browser developer tools, and provides a hands‑on Python example that fetches a page with the requests library and prints its source code.
Learn to quickly build a Python web scraper using requests and Xpath to extract Douban movie 'Black Panther' short reviews, covering setup, HTTP request analysis, data parsing, storage with pandas, and best practices like polite crawling intervals, all demonstrated with concise 12-line code.
This tutorial walks you through using Python's requests and XPath to scrape short comments from Douban's "Black Panther" page, covering tool setup, HTTP request analysis, data extraction, and saving results to CSV in just a dozen lines of code.
This step‑by‑step tutorial shows Python beginners how to create a simple web scraper that downloads, parses, and stores the Maoyan movie TOP 100 list using requests, regular expressions, and multiprocessing for fast data collection.
This tutorial explains how to collect the names and trading information of every stock listed on the Shanghai and Shenzhen exchanges using Python 3.5, requests, BeautifulSoup and regular expressions, then store the results in a file through a clear three‑step process of fetching the stock list, retrieving each stock's page, parsing the HTML and writing the data.
This tutorial walks through building a Python web scraper that automatically downloads images from Baidu by analyzing requirements, inspecting page source, crafting regex patterns, and implementing the crawler with requests, offering step‑by‑step guidance, code snippets, and troubleshooting tips.
This article walks through building a Python web scraper for Zhihu, covering login simulation, extracting user profiles, answer likers, followers, avatars, and all answers of a question, and storing the collected data in SQLite, while highlighting challenges like captcha and anti‑scraping limits.
This article introduces five highly practical Python packages—yagmail, requests, psutil, BeautifulSoup, and a collection of utility scripts—explaining how each simplifies common tasks such as sending emails, making HTTP calls, system monitoring, web scraping, and code reuse, complete with concise code examples.
This guide compiles eleven practical Python requests tips—including preserving cookies across calls, simplifying headers, adding retry logic, disabling redirects, sending JSON payloads, extracting debug information, performing asynchronous requests with grequests, customizing cookies, mocking APIs, measuring request time, and setting timeouts—to help developers write cleaner, more reliable HTTP code.
The article describes a Python requests pitfall where a JSON‑encoded parameter triggers automatic URL encoding causing request failures, explains how to work around it, and then shares a small PyPDF2 script used to split a large PDF into separate pages.
This tutorial walks you through setting up a macOS environment, inspecting a web page, and writing a Python script with the requests library to locate and download all images from a target site, complete with code explanations and execution tips.
