0 views collected around this technical thread.
This article outlines essential Python web‑crawling techniques—including basic GET/POST requests, proxy usage, cookie management, header spoofing, captcha handling, gzip compression, and multithreaded fetching—to help developers build efficient and robust crawlers.
This article describes how to build a Python web crawler that automatically generates numeric URLs, checks their validity, extracts download links for TV dramas from cn163.net, saves them to text files, and discusses practical challenges such as regex parsing, filename restrictions, and multithreading performance.
This article explains why web servers block crawlers, how to identify a browser's User-Agent (using Chrome as an example), and demonstrates step‑by‑step how to disguise a scraper as a browser to retrieve the CSDN homepage and its article list.
This article introduces eight widely used Python web scraping frameworks—including Scrapy, PySpider, Crawley, Portia, Newspaper, Beautiful Soup, Grab, and Cola—describing their main features, typical use cases, and providing links to their project repositories.
This article provides a comprehensive overview of search engine fundamentals—including inverted indexing, tokenization, ranking, high‑concurrency infrastructure, caching, crawling strategies, query understanding, keyword rewriting, personalization, and knowledge‑base construction—highlighting the technical challenges that make modern search engines like Google superior to simpler implementations.
This article provides a comprehensive overview of search engine evolution, core technologies such as crawling, indexing, retrieval and link analysis, platform foundations including cloud storage and computing, and techniques for improving search results through anti‑spam, user‑intent analysis, deduplication and caching.