How to Crawl CSDN Geek News Articles with Python: A Step‑by‑Step Guide

1. Idea

The goal is to scrape the latest articles from CSDN's Geek News section after reading the book "Python Web Data Collection"; the process involves logging in, requesting the article list, and parsing the returned JSON to obtain titles and URLs.

2. Preparation

By inspecting network traffic, the request URL for fetching a new list is:

http://geek.csdn.net/service/news/get_news_list?jsonpcallback=jQuery203014439105321047596_1516862462757&username=[username]&from=-&size=20&type=hackernewsv2_new&_=1516862462758

The relevant parameters are:

jsonpcallback : automatically generated by jQuery, can be any string.

username : your CSDN account name.

from : the starting index for the next request; use "-" for the first request.

size : number of articles to retrieve (e.g., 20, even 1000 works).

type : article category such as hackernewsv2_new.

_ : a timestamp-like number, optional.

Login requires only username and password; the request includes additional fields:

gps : geographic coordinates.

rememberMe : true to keep the session.

lt and execution : tokens extracted from the login page.

_eventId : fixed value "submit".

When logging in, the HTTP User-Agent header must mimic a real browser, otherwise CSDN blocks the request as a bot.

3. Code (very short)