Python Web Scraping Basics: Extract Strings and Intro to Regex in 5 Minutes

1. Pre‑review and Common Errors

Before coding, the article lists typical installation and URL‑format errors gathered from 13 documents. It notes that the USTC mirror requires browser verification, while Douban and Alibaba mirrors often raise "webpage parsing failed"; the Tsinghua mirror works without verification. It also explains that extra characters in URLs (quotes, newlines, trailing commas) cause parsing failures.

# Install requests via Tsinghua mirror
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple requests

# Correct request example
import requests
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"}
url = "https://httpbin.org/get?keyword=Python"
response = requests.get(url, headers=headers, timeout=5)
print(response.status_code)  # 200 = success

2. Core Method 1 – String Extraction

The article demonstrates using Python's built‑in str.find() and slicing to pull values directly from the response text, avoiding additional JSON parsing when the request fails.

import requests
headers = {"User-Agent": "..."}
url = "https://httpbin.org/get?keyword=Python"
response = requests.get(url, headers=headers, timeout=5)
response.raise_for_status()
text = response.text

# Extract client IP
ip_start = text.find('"origin": "') + len('"origin": "')
ip_end = text.find('"', ip_start)
ip = text[ip_start:ip_end]
print("Client IP:", ip)

# Extract keyword if present
if "keyword" in text:
    kw_start = text.find('"keyword": "') + len('"keyword": "')
    kw_end = text.find('"', kw_start)
    keyword = text[kw_start:kw_end]
    print("Keyword:", keyword)

# Extract full request URL
url_start = text.find('"url": "') + len('"url": "')
url_end = text.find('"', url_start)
request_url = text[url_start:url_end]
print("Request URL:", request_url)

except Exception as e:
    print(f"Request/parse error: {e}")
    if "网页解析失败" in str(e) or "不支持的网页类型" in str(e):
        print("Solution: 1) Check URL for extra symbols; 2) Ensure no trailing commas/spaces; 3) Use string extraction as fallback")

3. Core Method 2 – Basic Regular Expressions

Using the built‑in re module, the article shows concise patterns to match IP, keyword and URL fields, handling cases where the response contains extra trailing ampersands.

import requests, re
headers = {"User-Agent": "..."}
url = "https://httpbin.org/get?keyword=Python入门"
response = requests.get(url, headers=headers, timeout=5)
response.raise_for_status()
text = response.text

# IP pattern
ip_pattern = r'"origin": "(\d+\.\d+\.\d+\.\d+)"'
ip = re.findall(ip_pattern, text)
if ip:
    print("Regex IP:", ip[0])

# Keyword pattern
keyword_pattern = r'"keyword": "([^"]+)"'
keyword = re.findall(keyword_pattern, text)
if keyword:
    print("Regex Keyword:", keyword[0])

# URL pattern
url_pattern = r'"url": "([^"]+)"'
request_url = re.findall(url_pattern, text)
if request_url:
    print("Regex URL:", request_url[0])

except Exception as e:
    print(f"Request/parse error: {e}")
    if "网页解析失败" in str(e):
        print("Solution: Check URL format, ensure no extra symbols, fall back to string extraction")

4. Integrated Practice – Batch Requests and Comparison

A complete script sends multiple correct URLs, extracts data with both string and regex methods, prints each result, and handles exceptions without terminating the loop.

import requests, re
from datetime import datetime
headers = {"User-Agent": "..."}
urls = [
    "https://httpbin.org/get?keyword=Python",
    "https://httpbin.org/get?keyword=爬虫",
    "https://httpbin.org/get?keyword=数据解析"
]
print(f"Start time: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
print('-' * 50)

for index, url in enumerate(urls, 1):
    try:
        response = requests.get(url, headers=headers, timeout=5)
        response.raise_for_status()
        text = response.text
        # String extraction
        ip_start = text.find('"origin": "') + len('"origin": "')
        ip_end = text.find('"', ip_start)
        ip_str = text[ip_start:ip_end]
        # Regex extraction for keyword
        keyword = re.findall(r'"keyword": "([^\"]+)"', text)[0]
        print(f"Result {index}:")
        print(f"  URL: {url}")
        print(f"  IP (string): {ip_str}")
        print(f"  Keyword (regex): {keyword}")
        print('-' * 50)
    except Exception as e:
        print(f"Result {index} failed (URL: {url}): {e}")
        if "网页解析失败" in str(e):
            print("  Hint: Remove extra symbols from URL and retry")
            print('-' * 50)

5. Common Pitfalls for Beginners

Installing requests from most mirrors fails – use the Tsinghua mirror.

Improper URL formatting (extra quotes, newlines, trailing commas) triggers "webpage parsing failed".

Regex patterns that do not match the actual JSON format return empty results.

Batch requests may contain malformed URLs; ensure every URL follows the documented format and wrap calls in try‑except blocks.

6. Key Takeaways

The core of web‑scraping data parsing is combining string extraction as a fallback with regex for precise matches.

Master str.find() slicing, the re module, and exception handling to cover all error cases documented in the 13 reference files.

Always use a reliable PyPI mirror (Tsinghua), clean URLs of extraneous characters, and prefer string extraction when JSON parsing fails.