This article explains web crawler fundamentals, from the need for automated data collection amid information explosion to the URL-based queue crawling process, basic Python libraries, and the role of frameworks in boosting development efficiency.
This tutorial shows how to build a JD.com search URL, fetch the page with urllib, parse the HTML using BeautifulSoup, and reliably extract each product's name, link, image and price while handling missing image URLs and avoiding regex complexity.
This tutorial shows how to build a JD.com search URL, encode keywords, fetch the page with Python's urllib, and extract product details using regular expressions, providing code snippets, regex explanations, and sample output for beginners.
This article introduces beginner-friendly Python web‑scraping methods, covering the simplest urllib/requests approach, adding request headers, inspecting network traffic for hidden data, handling dynamically loaded content with Selenium, and provides links to deeper tutorials for each technique.
This tutorial walks through multiple Python techniques for downloading regular files, web pages, Amazon S3 objects, and other resources, covering basic requests, wget, handling redirects, chunked large‑file downloads, parallel downloads, progress bars, urllib, urllib3, proxy usage, boto3 for S3, and asynchronous downloads with asyncio.
This tutorial teaches how to download files in Python using various modules such as requests, wget, urllib, urllib3, boto3, and asyncio, covering basic downloads, handling redirects, chunked large-file downloads, parallel batch downloads, progress bars, proxy usage, and asynchronous techniques.
This tutorial teaches how to download files in Python using various modules such as requests, wget, urllib, urllib3, boto3 for S3, and asyncio, covering basic downloads, redirects, chunked large‑file handling, parallel downloads, progress bars, proxy usage, and asynchronous techniques.
This article outlines essential Python web‑scraping techniques, covering basic GET/POST requests, proxy usage, cookie handling, header manipulation to mimic browsers, simple captcha solutions, gzip compression handling, and multithreaded crawling with a thread‑pool template, providing practical code examples for each step.
This tutorial explains how to download regular files, web pages, Amazon S3 objects, and other resources in Python by covering multiple modules such as requests, wget, urllib, urllib3, Boto3, and asyncio, and demonstrates handling redirects, large files, parallel downloads, progress bars, proxies, and asynchronous execution.
This article introduces Python's urllib module, explains its main submodules—urllib.request, urllib.parse, urllib.error, and urllib.robotparser—and provides practical code examples demonstrating URL opening, parsing, error handling, and robots.txt processing for interface automation tasks in Python.
This tutorial demonstrates multiple Python techniques for downloading files—including simple requests.get calls, wget and urllib modules, handling redirects and large files with chunked streaming, parallel batch downloads, progress bars, proxy support, S3 retrieval via boto3, and asynchronous downloads with asyncio—providing a comprehensive guide for developers.
This article provides a detailed tutorial on Python's urllib library, covering its main modules (request, error, parse, robotparser), key functions and classes, code examples for URL fetching, parsing, encoding, and handling robots.txt, making it a practical resource for backend developers and web scrapers.
This tutorial walks through multiple Python approaches for downloading files—including simple requests.get calls, the wget module, handling redirects, chunked large‑file downloads, parallel batch downloads, proxy usage with urllib, S3 retrieval via Boto3, and asynchronous fetching with asyncio—providing code examples and best‑practice tips.
An in‑depth guide shows how to decode URL‑encoded strings encountered during Python web scraping, explains the difference between two encoding formats, and provides ready‑to‑run urllib code that prints the original Chinese characters, helping developers troubleshoot similar crawling issues.
This article explains how to use Python's urllib library to decode URL‑encoded strings encountered in web crawling, walks through a real example with code, and shows the resulting decoded URL, helping developers troubleshoot common encoding issues.
This article explains how to build a Python web crawler that fetches HTML pages with urllib, parses them using BeautifulSoup to extract image URLs, and downloads the images, covering all three stages with complete code examples and parser options.
This article demonstrates two Python approaches—using high-level requests and low-level HTTPConnection—to fetch intraday stock data from Eastmoney, providing complete code examples, URL discovery steps, and practical tips for data‑mining enthusiasts.
This article demonstrates how to automate Zhihu login on Windows by analyzing network traffic with Fiddler, extracting required parameters, and implementing a Python script that builds HTTP requests using urllib2, handles cookies, captcha retrieval, and logs the results, complete with sample code and execution screenshots.
This article introduces Python 3 methods for retrieving various document types—including TXT, CSV, PDF, and Word files—using urllib, regular expressions, and file‑specific processing steps, providing practical code examples and workflow guidance for building effective web crawlers.
This article compares Python's built‑in urllib library with the third‑party requests library, demonstrating their usage through code examples, highlighting differences in request construction, response handling, and practical considerations for web scraping, and concludes with recommendations for choosing the more convenient tool.
This tutorial explains how to download files, webpages, and Amazon S3 objects using Python, covering the requests and wget modules, handling redirects, chunked and parallel downloads, progress bars, proxy support, urllib/urllib3 usage, and asynchronous downloading with asyncio.
This tutorial guides readers through analyzing web pages, using Python to write network programs, applying regular expressions, and operating Excel to scrape all listed company stock data for a specified date range, save it by stock code, and optionally store results in databases.
This tutorial walks through using Python's urllib and BeautifulSoup libraries to fetch, parse, clean, and export the Fast Track 100 company table into a CSV file, covering installation, page inspection, element extraction, data cleaning, link handling, and file writing.
This tutorial demonstrates how to use Python's urllib module to simulate browser requests, parse HTML with Beautiful Soup, extract text and image URLs, and store the scraped data locally using file I/O and the with statement, providing complete code examples.
Learn how to use Python’s built-in urllib library for web requests, handling GET/POST, adding headers, managing proxies, processing cookies, handling errors, parsing URLs, and respecting robots.txt, with clear code examples and a practical case of scraping a novel site.
This tutorial walks you through using various Python modules—including requests, wget, urllib, urllib3, boto3, and asyncio—to download regular files, web pages, Amazon S3 objects, handle redirects, large files, parallel downloads, proxies, and progress bars, all with clear code examples.
This tutorial shows how to use Python's urllib and json libraries to call the Kuaidi100 API, retrieve real‑time logistics data for various Chinese courier companies, and display the tracking timeline, while explaining how to discover the correct request URL via browser dev tools.
This tutorial walks through multiple Python approaches for downloading files—including simple requests and wget calls, handling redirects, large and multi‑file downloads, proxy usage, urllib/urllib3 methods, and asynchronous techniques—providing complete code snippets and practical tips for each scenario.
This comprehensive guide walks you through Python web crawling fundamentals, covering request handling, URL encoding, regular expressions, the requests library, XPath parsing, and lxml, complete with code snippets and practical examples to help you build effective scrapers.
This guide explains the fundamentals of web crawling, covering crawler types, the Robots.txt protocol, Python's urllib and urllib3 modules, the requests library, handling HTTP methods, user‑agents, HTTPS certificates, and practical code examples for extracting data from websites.
Learn how to start web scraping with Python by mastering the three core steps—fetching, analyzing, and storing data—using urllib and requests, handling login, evading anti‑scraping measures like user‑agents and IP proxies, and saving results to JSON, CSV, or MongoDB.
Learn step-by-step how to use Python's urllib and regular expressions to fetch Baidu Tieba web pages, extract image URLs, and download the images to your local machine, including code examples for getHtml and getImg functions and file naming conventions.
This tutorial explains how to use Python's urllib module to fetch a Baidu Tieba page, apply regular expressions to extract image URLs, and then download and rename those images to your local directory.
This tutorial shows how to build a keyword‑driven web crawler for JD.com using Python's urllib for URL encoding and opening, combined with powerful regular expressions to accurately extract product information such as dog food listings, and explains how to extend the scraper for multi‑page data collection.
This tutorial walks through using Python's urllib and regular expressions to crawl a Baidu Tieba page, extract all .jpg image URLs, and download each image locally with a sequential naming scheme.
Learn to use Python's urllib and regular expressions to fetch Baidu Tieba web pages, extract image URLs, and download the images to your local machine, with step-by-step code examples and explanations for beginners tackling web scraping challenges.
The article explains how Python's urllib2 raises URLError and HTTPError, details their causes and attributes, and presents two practical approaches for catching and processing these HTTP exceptions to build more robust networked applications.
This article explains how Python's urllib and urllib2 modules differ and demonstrates practical techniques for sending HTTP requests, encoding query strings, setting custom headers, handling JSON payloads, managing cookies, and configuring proxies using Request objects and build_opener handlers.
