How to Scrape All News from Sichuan University Public Administration Site with Python

Task

Scrape all news and announcements from the Sichuan University Public Administration College website (http://ggglxy.scu.edu.cn).

Experiment Process

1. Determine the scraping target. 2. Define scraping rules. 3. Write/debug the scraping rules. 4. Retrieve the scraped data.

1. Determine the Scraping Target

The goal is to collect every news item from the college’s website, which requires understanding the site’s layout.

All news cannot be captured from the homepage; the "more" button must be clicked to reach the news list page.

The news list page only provides time, title, and URL, so we need to follow each link to the detail page to obtain the full content.

2. Define Scraping Rules

Based on the analysis, the rule is: for each news link in the list, open the detail page and extract title, time, content, and URL.

To capture all news, we can use the "next page" button at the bottom of the list to iterate through pages.

3. Write/Debug Scraping Rules

Implementation focuses on three functions:

1. Crawl all news links on a list page. 2. Visit each news detail page to extract required data. 3. Loop through all pages to collect every article.

The corresponding knowledge points are:

1. Extract basic data from a page. 2. Perform secondary crawling based on the first‑level data. 3. Use loops to crawl the entire site.

3.1 Crawl All News Links on One Page

The news items are contained in the li.newsinfo_box_cf element, which can be selected and iterated.

Code Example

Test passed.

3.2 Extract Details from Each News Page

After obtaining the URLs, we fetch each detail page to extract title, time, and content using a simple request call.

Code Example

Integrating this into the main script yields the full crawler.

Test passed.

Adding a loop to process all pages:

After adjusting for two hidden secondary categories, the crawler captures all 193 news items.

Special handling for these secondary URLs is added to the rule set.

4. Retrieve Scraped Data

The final dataset contains title, publication time, URL, and full content for each news article.