Scrapy Tutorial: Crawling Comic Images with BeautifulSoup and Saving Locally
This article provides a step‑by‑step guide on configuring Scrapy, creating a spider project, extracting comic page URLs and images using BeautifulSoup, handling pagination, and saving the downloaded images locally with Python code.
Scrapy is a Python framework for extracting structured data from websites; this tutorial demonstrates how to use it to crawl comic images.
Scrapy environment configuration
Install the required packages on macOS:
pip install Scrapypip install beautifulsoup4pip install html5libVerify installation by running scrapy and checking the output.
Project creation
Create a new Scrapy project named Comics: scrapy startproject Comics The generated directory structure looks like:
|____Comics| |______init__.py| |______pycache__| |______items.py| |______pipelines.py| |______settings.py| |______spiders| | |______init__.py| | |______pycache__|____scrapy.cfgPrint the structure with: find . -print | sed -e 's;[^/]*/;|____;g;s;____|; |;g' Creating the Spider class
In Comics/spiders/comics.py define a spider that inherits from scrapy.Spider:
#coding:utf-8import scrapyclass Comics(scrapy.Spider):name = "comics"def start_requests(self):urls = ['http://www.xeall.com/shenshi']for url in urls:yield scrapy.Request(url=url, callback=self.parse)def parse(self, response):self.log(response.body);Run the spider with: scrapy crawl comics The spider prints log information and the HTML source of the target page.
Crawling comic URLs
In parse, use BeautifulSoup to extract the list of comics:
from bs4 import BeautifulSoupcontent = response.body;soup = BeautifulSoup(content, "html5lib")listcon_tag = soup.find('ul', class_='listcon')com_a_list = listcon_tag.find_all('a', attrs={'href': True})comics_url_list = []base = 'http://www.xeall.com'for tag_a in com_a_list:url = base + tag_a['href']comics_url_list.append(url)Handle pagination by locating the ul.pagelist element, extracting the next‑page URL, and yielding a new request unless the current page is the last one:
page_tag = soup.find('ul', class_='pagelist')page_a_list = page_tag.find_all('a', attrs={'href': True})select_tag = soup.find('select', attrs={'name': 'sldd'})option_list = select_tag.find_all('option')last_option = option_list[-1]current_option = select_tag.find('option', attrs={'selected': True})is_last = (last_option.string == current_option.string)if not is_last:next_page = 'http://www.xeall.com/shenshi/' + page_a_list[-2]['href']yield scrapy.Request(next_page, callback=self.parse)For each comic URL, schedule a request to self.comics_parse:
for url in comics_url_list:yield scrapy.Request(url=url, callback=self.comics_parse)Extracting comic images
In comics_parse, parse the page and locate the image tag:
def comics_parse(self, response):content = response.body;soup = BeautifulSoup(content, "html5lib")li_tag = soup.find('li', id='imgshow')img_tag = li_tag.find('img')img_url = img_tag['src']title = img_tag['alt']Save the image locally using a helper method: self.save_img(page_num, title, img_url) Saving images to disk
# 先导入库import osimport urllibimport zlibdef save_img(self, img_mun, title, img_url):self.log('saving pic: ' + img_url)document = '/Users/moshuqi/Desktop/cartoon'comics_path = document + '/' + titleif not os.path.exists(comics_path):os.makedirs(comics_path)pic_name = comics_path + '/' + img_mun + '.jpg'if os.path.exists(pic_name):self.log('pic exists: ' + pic_name)returntry:user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'headers = { 'User-Agent' : user_agent }req = urllib.request.Request(img_url, headers=headers)response = urllib.request.urlopen(req, timeout=30)data = response.read()if response.info().get('Content-Encoding') == 'gzip':data = zlib.decompress(data, 16 + zlib.MAX_WBITS)fp = open(pic_name, "wb")fp.write(data)fp.closeself.log('save image finished:' + pic_name)except Exception as e:self.log('save image error.')self.log(e)The spider continues to request the next image page until the # link indicates the last page.
Running results
When executed, the console shows log messages for each request, and the downloaded images are stored in folders named after each comic title, with filenames corresponding to page numbers. Scrapy runs multiple requests concurrently, so several comics are crawled in parallel.
Note that the target site may be slow, causing occasional timeouts; patience is required.
The article mentions that more advanced features such as FilesPipeline, ImagesPipeline, or using Scrapy’s built‑in XPath selectors can further improve efficiency.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
