Skip to content
How to Scrape Quotes from quotes.toscrape.com with Python and Save to CSV

How to Scrape Quotes from quotes.toscrape.com with Python and Save to CSV

This guide walks through using Python's requests and lxml libraries to scrape quotes, authors, and tags from quotes.toscrape.com, parse the HTML with XPath, and store the results in a CSV file, covering request headers, element inspection, and data extraction steps.

MaGe Linux Operations
MaGe Linux Operations
MaGe Linux Operations
Scraping illustration
Scraping illustration

1. Input the URL

Open https://quotes.toscrape.com/ and observe that the page contains three main fields: the quote text, the author, and tags. These are the data we want to extract.

2. Determine requirements and analyze the page structure

Use the browser’s developer tools (Network tab) to see that the site uses simple GET requests without parameters. We can therefore simulate the request with requests.get(), adding appropriate User‑Agent headers to avoid being blocked as a bot.

3. Parse the page and extract data

After a successful request, parse the HTML with lxml.etree and use XPath to locate the list of quotes. For each quote, extract the text, author, and associated tags, and store them in a Python list.

4. Save the data to a CSV file

Write the collected information into a CSV file for later analysis.

import requests
from lxml import etree
import csv

url = "https://quotes.toscrape.com/"
headers = {
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"
}

res = requests.get(url, headers=headers).text
html = etree.HTML(res)

quote_list = html.xpath('//div[@class="col-md-8"]')
lists = []

for quote in quote_list:
    # Quote text
    title = quote.xpath('./div[@class="quote"]/span[1]/text()')
    # Author
    author = quote.xpath('./div[@class="quote"]/span[2]/small/text()')
    # Tags
    tags = quote.xpath('./div[@class="quote"]/div[@class="tags"]/a[@class="tag"]/text()')
    lists.append(title)
    lists.append(author)
    lists.append(tags)

with open("./quotes.csv", "w", encoding="utf-8", newline="
") as f:
    writer = csv.writer(f)
    for item in lists:
        writer.writerow(item)

The above script demonstrates a complete workflow: sending a GET request with headers, parsing the response using XPath, extracting the desired fields, and persisting them to a CSV file.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Pythonweb scrapingXPath
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.