0 views collected around this technical thread.
This guide explains how unemployed developers can use Python, especially web‑scraping techniques, to secure freelance gigs by leveraging various platforms, community groups, and effective order‑taking strategies while warning about common pitfalls and anti‑scraping challenges.
The article explains several typical situations—such as not configuring HTTPS proxies, using server IPs, non‑anonymous proxies, polluted IP pools, and lack of HTTP/2 support—that cause websites to easily detect that a request is made through a proxy, even for beginner Python scrapers.
This article explains how to overcome common anti‑scraping defenses such as identity verification and IP rate limiting by spoofing the User‑Agent header and adding request delays, providing complete Python code examples using requests and BeautifulSoup to scrape Douban's Top 250 movies.
This tutorial explains how to analyze the Mayi short‑rent website, overcome its anti‑scraping defenses by setting appropriate Cookie and User‑Agent headers, and use Python's urllib2 and BeautifulSoup to extract rental details, store them in CSV, and optionally employ Selenium.
This article explains how to disguise a web crawler's identity by customizing request headers, managing request frequency with sleep and proxy settings, and tackling common anti‑scraping mechanisms such as captchas, dynamic loading, and encrypted content using tools like Selenium.
This article explains how many Chinese web sites use hidden CSS ::before content to hide characters, shows how to locate the relevant network request, decode the span class mappings from obfuscated JavaScript, and restore the original text for successful web scraping.
This article explains the concept and benefits of CSS sprites, analyzes their drawbacks for web performance and security, and provides a step‑by‑step Python‑based method—including code snippets—to extract and sum numbers hidden behind sprite images used as an anti‑scraping measure.
This article explains how to hide real intentions behind visible actions by using text obfuscation and custom Base64‑like encoding to defeat standard web scrapers, detailing the underlying principles, decoding challenges, and Python implementations of a flexible Custom64 encoder.
This guide explains practical techniques to avoid IP bans and 403 errors when web‑scraping, covering explicit and implicit waiting, User‑Agent spoofing, proxy usage, IP pools, and distributed crawling architectures.
This guide explains practical techniques such as faking User‑Agent headers, rotating cookies, adding random delays, and using proxy pools to prevent IP bans while crawling large amounts of data from websites with anti‑scraping defenses.
The ISC 2018 conference highlighted the growing importance of big data and artificial intelligence security, presenting JD's research on anti‑scraping techniques, AI‑driven defenses against black‑market attacks, and a service‑oriented approach to protecting user data across enterprises.