Master Web Scraper: Build Complete Chrome-Based Crawlers Without Code

1. Install Web Scraper

Download the Web Scraper plugin (search "webscraper"), unzip the package, then open Chrome’s Extensions page (More tools → Extensions) and click "Load unpacked" to import the folder. Enable the extension to see its icon on the toolbar.

2. Build a Complete Scraping Workflow

Create a new sitemap by clicking "Create new sitemap", set a name (e.g., "csdn") and a start URL such as https://www.csdn.net/. Save the sitemap.

After the sitemap is created, add selectors that define what data to extract. Each selector requires an ID, a Type (Text, Link, Image, Element, etc.), and a CSS selector that highlights the target element on the page.

Common selector options include:

ID : a name for the selector.

Type : the kind of data to capture (Text, Link, Element, etc.).

Selector : the element on the page to click or scrape.

Multiple : whether to capture multiple matching elements.

Regex : a regular expression to further filter captured text.

Delay : wait time before the selector runs.

Parent selectors : build hierarchical relationships between selectors.

3. Multi‑Level Data Extraction

To keep title and author data aligned, create a parent selector of type Elements that captures the whole article block, then add child selectors for the title and author within that block.

For deeper pages, first create a selector that captures the article link (type Link ), then add a child selector under it to scrape the article’s detail page (title, content, etc.).

4. Pagination Design Pattern

When a site changes the URL for each page, configure the start URL to iterate over a range (e.g., pages 1‑100). For sites without a predictable URL, add a selector that captures the “next page” link and set it as a parent of the element selector, creating a looping structure.

5. Infinite‑Scroll Design Pattern

For pages that load more data when scrolling, set the selector type to elements scroll down . The rest of the configuration remains the same, enabling automatic scroll‑triggered data capture.

6. Click‑Load Design Pattern

When a “Load more” button must be clicked, create a selector of type elements click . Specify the button element as the click selector and choose the click mode (click once or click more). Configure uniqueness rules to stop when the button disappears.

After configuring selectors, start the scrape. The tool will respect request intervals (default 2000 ms) and page‑load delays (default 500 ms). When scraping finishes, export the data as a CSV file.

These patterns cover the majority of small‑scale web‑scraping scenarios. For large‑scale crawling that encounters anti‑scraping measures, a custom coded solution may be required.