Getting Started with Web Crawlers: Inspect Elements and Simple Python Requests Demo

Introduction to Web Crawlers

Web crawlers, also called spiders, retrieve web page content by requesting URLs such as https://www.baidu.com/.

1. Inspect Elements

Open a page, right‑click and choose “Inspect” (Chrome) or “View Element” (Firefox) to see the HTML code returned by the server.

The HTML defines the page’s original appearance, similar to how genes determine a person’s look.

By editing the HTML in the developer tools you can locally “cosmetically” change a page, but changes are not saved on the server and disappear after a refresh.

Example: changing a password field’s type from password to text reveals the hidden password.

2. Simple Python Example

The first step of a crawler is to fetch the HTML of a URL. In Python 3 you can use the built‑in urllib.request module or the third‑party requests library.

Installation

pip install requests

easy_install requests

Basic usage

# -*- coding:UTF-8 -*-
import requests

if __name__ == '__main__':
    target = 'http://gitbook.cn/'
    req = requests.get(url=target)
    print(req.text)

This script sends a GET request to the target URL and prints the returned HTML.