What Is a Web Spider? Understanding URLs, URIs, and How Crawlers Work

1. Definition of a Web Crawler

A web crawler (Web Spider) is a program that traverses the Internet by following hyperlinks, starting from a seed page (often the homepage) and recursively fetching linked pages until the entire site is captured.

2. How a Browser Retrieves a Page

When a user enters a URL such as www.baidu.com, the browser acts as a client, sends an HTTP request to the server, receives the HTML file, parses it, and renders the page for the user.

3. URI and URL Concepts

A URI (Uniform Resource Identifier) uniquely identifies any resource on the web, consisting of three parts: a naming mechanism, the host name, and the resource path. Example: http://www.why.com.cn/myhtml/html1223/.

A URL (Uniform Resource Locator) is a subset of URI that also specifies the protocol used to access the resource.

4. URL Structure

The general URL format is: protocol://hostname[:port]/path[;parameters][?query]#fragment. It comprises three mandatory parts: the protocol, the host (and optional port), and the resource path. The first two parts are separated by "://" and the host and path are separated by "/".

5. URI vs. URL Comparison

URI is the abstract identifier for a resource, while URL is a concrete locator that tells how to retrieve that resource (e.g., using http://). Understanding the difference helps in correctly handling web resources.

Accurate comprehension of URLs is essential for web crawlers, as they rely on URLs to fetch and process web content.