Build a Super‑Simple Node.js Web Scraper for Douban Top 250

The article shows how to write a very simple web crawler in Node.js to scrape the Douban Top 250 movies list. It starts by initializing a project with npm init to generate a package.json file, then installs the cheerio library via npm i cheerio for DOM parsing.

Using Node's built‑in https module, the script sends a GET request to https://movie.douban.com/top250. The response data is concatenated into a string variable html inside the data event handler, and the end event signals that the full page has been received.

const https = require('https');
https.get('https://movie.douban.com/top250', function (res) {
  let html = '';
  res.on('data', function (chunk) { html += chunk; });
  res.on('end', function () {
    // parse and process html here
  });
});

After the page is fully loaded, cheerio.load(html) creates a jQuery‑like selector $. The script iterates over each movie item with $('li .item').each(...), extracting the title ( '.title'), rating ( '.info .bd .rating_num') and poster image URL ( '.pic img' attribute src). These values are stored in an object and pushed into an array allFiles.

const $ = cheerio.load(html);
let allFiles = [];
$('li .item').each(function () {
  const title = $('.title', this).text();
  const star = $('.info .bd .rating_num', this).text();
  const pic = $('.pic img', this).attr('src');
  allFiles.push({ title, star, pic });
});

The array is then written to files.json using Node's fs module. The script creates the file (or overwrites it) with fs.writeFile, serializing the array via JSON.stringify and logging a success message.

const fs = require('fs');
fs.writeFile('./files.json', JSON.stringify(allFiles), function (err) {
  if (err) throw err;
  console.log('文件保存成功');
});

Running the complete script produces a files.json file in the project directory containing an array of objects, each with the movie title, rating, and poster URL. The article concludes that writing a Node.js crawler is straightforward and useful for front‑end beginners learning back‑end JavaScript capabilities.