How to Scrape a Full Novel with Beautiful Soup: Step‑by‑Step Guide

1. Beautiful Soup Overview

Beautiful Soup is a third‑party Python library that simplifies extracting data from HTML or XML documents. It provides easy functions for traversing, searching, and modifying the document tree, handling encoding automatically and allowing concise crawler code.

Key Features

Extract data from HTML/XML, with simple traversal and search capabilities.

Handles Unicode conversion and outputs UTF‑8 without manual encoding work.

2. Installation

Install the library via the command line:

pip install beautifulsoup4

3. Basic Concepts

Important object types include Tag , NavigableString , BeautifulSoup , and Comment . Common methods for navigating the tree are find, find_all, find_next, and children. Basic HTML/CSS knowledge is helpful but not required.

4. Single‑Chapter Scraper

The target site (136book.com) lists chapters on a directory page. By inspecting the page structure, each chapter link resides in <li> elements inside a <div id="book_detail" class="box1">. The scraper fetches the chapter URL, sends a request with a custom User‑Agent header, reads the response, parses it with Beautiful Soup, and extracts the text inside the <div id="content"> element.

from urllib import request
if __name__ == '__main__':
    # URL of the chapter
    url = 'http://www.136book.com/huaqiangu/ebxeew/'
    head = {}
    # Set User-Agent header
    head['User-Agent'] = 'Mozilla/5.0 (Linux; Android 4.1.1; Nexus 7 Build/JRO03D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Safari/535.19'
    req = request.Request(url, headers=head)
    response = request.urlopen(req)
    html = response.read()
    soup = BeautifulSoup(html, 'lxml')
    soup_text = soup.find('div', id='content')
    print(soup_text.text)

The script prints the chapter content, confirming successful extraction.

5. Full‑Book Scraper

To download the entire novel, first parse the directory page to collect all chapter links, then iterate over each link, fetch its content, and write both the chapter title and text to a local file (e.g., F:/huaqiangu.txt). The process reuses the same request headers and parsing logic, adding file handling for output.

from urllib import request
from bs4 import BeautifulSoup
if __name__ == '__main__':
    url = 'http://www.136book.com/huaqiangu/'
    head = {'User-Agent': 'Mozilla/5.0 (Linux; Android 4.1.1; Nexus 7 Build/JRO03D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Safari/535.19'}
    # Parse directory page
    req = request.Request(url, headers=head)
    response = request.urlopen(req)
    html = response.read()
    soup = BeautifulSoup(html, 'lxml')
    # Locate the second <div id="book_detail" class="box1"> containing the full list
    soup_texts = soup.find('div', id='book_detail', class_='box1').find_next('div')
    f = open('F:/huaqiangu.txt', 'w')
    for link in soup_texts.ol.children:
        if link != '':
            download_url = link.a.get('href')
            download_req = request.Request(download_url, headers=head)
            download_response = request.urlopen(download_req)
            download_html = download_response.read()
            download_soup = BeautifulSoup(download_html, 'lxml')
            download_soup_texts = download_soup.find('div', id='content')
            # Write chapter title
            f.write(link.text + '
')
            # Write chapter content
            f.write(download_soup_texts.text)
            f.write('
')
    f.close()

The script saves the complete novel to the specified file, and the execution log shows successful completion.

6. Summary

The tutorial demonstrates how to use Beautiful Soup for both single‑chapter and full‑novel scraping, highlighting the importance of analyzing page structure, handling HTTP headers, and iterating over links. Further improvements could include removing embedded ads with regular expressions, adding progress indicators, and using additional modules such as os for file management.