Master Web Scraping with Beautiful Soup: A Hands‑On Python Guide

Beautiful Soup Overview

Beautiful Soup is a Python library for extracting data from HTML or XML files. It parses the markup into a tree structure, making it easy to locate tags and retrieve their attributes.

The library allows you to specify a class or id as a parameter to directly fetch the corresponding tag data.

The latest version is Beautiful Soup 4 (4.4.0); Beautiful Soup 3 is no longer maintained. BS4 works with Python 2.7 and Python 3.x.

Installation

sudo easy_install beautifulsoup4

After installation, import the library to verify it works:

from bs4 import BeautifulSoup

Creating a BeautifulSoup Object

You can initialize a BeautifulSoup object by passing a document string to its constructor. The example below fetches a page via urllib2 and parses it:

#coding:utf-8
from bs4 import BeautifulSoup
import urllib2
url = 'http://reeoo.com'
request = urllib2.Request(url)
response = urllib2.urlopen(request, timeout=20)
content = response.read()
soup = BeautifulSoup(content, 'html.parser')

Alternatively, you can initialize from a local file:

soup = BeautifulSoup(open('reo.html'))

Working with Tags

Access a tag directly:

tag = soup.title
print tag

Output:

<title>Reeoo - web design inspiration and website gallery</title>

Retrieve the tag name: print tag.name # title Access attributes (similar to a dictionary):

tag = soup.article
c = tag['class']
print c  # [u'box']
attrs = tag.attrs
print attrs  # {u'class': [u'box']}

Get the string inside a tag:

tag = soup.title
s = tag.string
print s  # Reeoo - web design inspiration and website gallery

Traversing the Document Tree

Navigate child nodes:

tag = soup.article.div.ul.li
print tag

Find all li tags under the ul: ls = soup.article.div.ul.find_all('li') List a tag's direct children:

tag = soup.article.div.ul
contents = tag.contents

Iterate over children using the .children generator:

tag = soup.article.div.ul
children = tag.children
for child in children:
    print child

Access parent nodes:

tag = soup.article
print tag.parent.name  # body

Iterate over all ancestors:

tag = soup.article
for p in tag.parents:
    print p.name

Searching the Tree

Use find_all() to locate tags. Common parameters include name, attrs, recursive, string, and others.

soup.find_all('title')
# returns a list with the <title> tag

soup.find_all(id='footer')
# finds tags with id="footer"

When a tag name collides with a Python keyword, append an underscore (e.g., class_). soup.find_all('div', class_='thumb') Search by attribute existence: soup.find_all(target=True) Search using a dictionary of attributes: print soup.find_all(attrs={'data-original': True}) Search by string content with regular expressions: soup.find_all(string=re.compile('Reeoo')) Limit the number of results: soup.find_all('div', class_='thumb', limit=3) Restrict search to direct children only:

soup.find_all('div', recursive=False)

Using find()

find()

works like find_all() but returns only the first match.

soup.find('div', class_='thumb')

CSS Selectors

BeautifulSoup also supports CSS selectors via select():

soup.select('article ul li')
# selects all <li> inside <ul> inside <article>

soup.select('.thumb')
# selects elements with class "thumb"

soup.select('#sponsor')
# selects element with id "sponsor"

soup.select('li[id]')
# selects <li> elements that have an id attribute

soup.select('li[id="sponsor"]')

Additional Search Methods

Other useful methods include find_parents(), find_parent(), find_next_siblings(), find_next_sibling(), find_previous_siblings(), and find_previous_sibling(). These provide fine‑grained control over navigation and are sufficient for most scraping tasks.

For detailed information, refer to the official Beautiful Soup documentation.