Master Web Scraping with Beautiful Soup: A Hands‑On Python Guide

Beautiful Soup is a Python library that parses HTML or XML into a tree structure, enabling easy extraction of tag attributes and text.

Install it on macOS with sudo easy_install beautifulsoup4 and import it using from bs4 import BeautifulSoup.

Getting Started

The tutorial demonstrates scraping http://reeoo.com :

Initialize a BeautifulSoup object by fetching the page with urllib2:

#coding:utf-8<br/>from bs4 import BeautifulSoup<br/>import urllib2<br/>url = 'http://reeoo.com'<br/>request = urllib2.Request(url)<br/>response = urllib2.urlopen(request, timeout=20)<br/>content = response.read()<br/>soup = BeautifulSoup(content, 'html.parser')

Alternatively, load a local HTML file:

soup = BeautifulSoup(open('reo.html'))

Tag

Access tags directly, e.g., tag = soup.title and print the tag.

Attributes

Retrieve a tag’s attributes like a dictionary: c = tag['class'] or attrs = tag.attrs. Note that multi‑value attributes such as class are returned as lists.

Tag strings

Get the text inside a tag with tag.string.

Traversing the Document Tree

Use .contents to get a list of a tag’s direct children, .children as a generator, .parent for the immediate parent, .parents to iterate over all ancestors, and .next_sibling / .previous_sibling for sibling navigation.

Searching the Tree

find_all()

Signature: soup.find_all(name, attrs, recursive, string, **kwargs). Examples include finding all title tags, tags with a specific id, filtering by class (using class_ because class is a Python keyword), using regular expressions, limiting results, and disabling recursion.

find()

Works like find_all but returns only the first matching element.

CSS selectors

Use soup.select('article ul li') to select li elements under article ul, soup.select('.thumb') for elements with class thumb, soup.select('#sponsor') for the element with id sponsor, and attribute selectors such as soup.select('li[id]') or soup.select('li[id="sponsor"]').

Other search methods

Additional helpers include find_parents, find_parent, find_next_siblings, find_next_sibling, find_previous_siblings, and find_previous_sibling, which behave similarly to the methods above.

For comprehensive details, refer to the official Beautiful Soup documentation.