BeautifulSoup Web Scraping with Python: Examples and Projects

BeautifulSoup parses HTML or XML so Python can extract webpage titles, headings, links, and metadata. Fetch a page with Requests, build a BeautifulSoup object, and select the elements you need.

 Copy
import requests
from bs4 import BeautifulSoup

url = "https://www.plus2net.com/html_tutorial/html-canvas.php"
response = requests.get(url, timeout=15)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
print(soup.title.get_text(strip=True) if soup.title else "No title")

The example fetches HTML, checks for HTTP errors, and prints the page title. The html.parser parser comes with Python.

Interactive BeautifulSoup Practice Notebook

Practice fetching HTML, extracting titles, headings and links, reading metadata, checking canonical URLs, and generating a table of contents with BeautifulSoup in Google Colab.

View on GitHub

Install BeautifulSoup and parsers 🔝

Install BeautifulSoup and Requests before running the examples. lxml is optional when you want an additional HTML/XML parser.

pip install beautifulsoup4 requests

If you use Anaconda, you can install BeautifulSoup through conda:

conda install -c conda-forge beautifulsoup4 requests

Optional parser for XML and alternate HTML parsing:

pip install lxml

Parser choices: html.parser is bundled with Python. lxml is usually faster for large documents and can parse XML, while html5lib follows HTML5-style parsing rules when installed. Parser choice can change how malformed HTML is interpreted.

BeautifulSoup web scraping video 🔝

Fetch webpage HTML with Requests 🔝

Before parsing, collect the webpage source using the Python requests library. The response contains the HTML returned by the server.

import requests
link = "https://www.plus2net.com/html_tutorial/html-canvas.php"
content = requests.get(link)
print(content.text)

The original short example prints the entire HTML. For a reusable scraper, specify a timeout, check the response status, and print only a preview while testing.

 Copy
import requests

url = "https://www.plus2net.com/html_tutorial/html-canvas.php"
try:
    response = requests.get(url, timeout=15)
    response.raise_for_status()
    print("HTTP status:", response.status_code)
    print(response.text[:300])
except requests.RequestException as err:
    print("Unable to fetch the page:", err)

Some websites render content with JavaScript after the HTML response arrives. Requests does not execute browser JavaScript, so you may only receive the initial HTML. Always follow the site's access rules and use a reasonable request rate.

Create a BeautifulSoup object 🔝

Pass the downloaded HTML and a parser name to BeautifulSoup(). The resulting soup object provides methods to search and traverse the HTML tree.

from bs4 import BeautifulSoup
soup = BeautifulSoup(content.text, 'html.parser')

For XML documents, use BeautifulSoup(xml_text, "xml") with lxml installed. For regular HTML tutorials, html.parser is a convenient starting point.

Extract page titles, headings and parent tags 🔝

BeautifulSoup exposes many common tags as attributes. The following example preserves the original approach to reading title and h1 while showing how to inspect their content and parent elements.

 Copy
import requests
link = "https://www.plus2net.com/html_tutorial/html-canvas.php"
content = requests.get(link)

from bs4 import BeautifulSoup
soup = BeautifulSoup(content.text, 'html.parser')
print(soup.title) # gets title tag
print(soup.h1)    # gets H1 tag
print(soup.h1.string)
#print(soup.title.parent) # full string within the parent tag
print(soup.title.parent.name)

Output depends on the current HTML of the target page. A title or heading containing child tags can return None for .string; get_text() is more reliable when you want all visible text inside an element.

title_tag = soup.find("title")
h1_tag = soup.find("h1")
print(title_tag.get_text(strip=True) if title_tag else "No title")
print(h1_tag.get_text(" ", strip=True) if h1_tag else "No H1")
print(title_tag.parent.name if title_tag else "No title parent")

Collect all H2 headings

h2_tags=soup.find_all('h2') # List of all h2 tags
h2_strings = [tag.string for tag in h2_tags] # remove the h2 tag and keep only string part

Use get_text(" ", strip=True) when headings contain nested elements such as a link or a span:

h2_titles = [tag.get_text(" ", strip=True) for tag in soup.find_all("h2")]
for title in h2_titles:
    print(title)

For deeper traversal, see BeautifulSoup find_all() and children and parent tags.

Parse HTML stored in a Python string 🔝

You do not need a live website to practice. BeautifulSoup also parses an HTML string already stored in Python. This is useful for debugging selectors without making HTTP requests.

 Copy
content = """<html>
<head>
<title>Your title of the page here</title>
<META NAME='DESCRIPTION' CONTENT='my description '>
<META NAME='KEYWORDS' CONTENT='kw1,kw2,kw3'>
</head>
<body>
Hello <br>
Welcome to plus2net.com
</body>
</html>"""

from bs4 import BeautifulSoup
soup = BeautifulSoup(content, 'html.parser')
print(soup.title) # gets title
print(soup.h1)    # gets H1 tag
print(soup.title.name) # name of the tag i.e title
print(soup.title.string)
print(soup.title.parent) # full string within the parent tag
print(soup.title.parent.name)

In this sample, soup.h1 prints None because no <h1> tag exists. The title element is inside the head element.

<title>Your title of the page here</title>
None
title
Your title of the page here
<head>
<title>Your title of the page here</title>
<meta content="my description " name="DESCRIPTION"/>
<meta content="kw1,kw2,kw3" name="KEYWORDS"/>
</head>
head

Handle missing tags without exceptions

If a tag is absent, accessing its .string raises an AttributeError. Check that the element exists first.

h2_tag = soup.find("h2")
print(h2_tag.get_text(strip=True) if h2_tag else "No H2 on this page")

Read meta tags and HTML attributes 🔝

Meta descriptions, keywords, Open Graph titles, and image attributes are stored as HTML attributes. The original sample can retrieve the two meta content values:

print(soup.meta['content']) # first meta tag, attribute content
print(soup.find_all('meta')[1]["content"]) # second meta tag

my description 
kw1,kw2,kw3

For pages where meta tags appear in a different order, select by attribute instead of array position:

meta_description = soup.find("meta", attrs={"name": "DESCRIPTION"})
meta_keywords = soup.find("meta", attrs={"name": "KEYWORDS"})
print(meta_description.get("content", "") if meta_description else "Not found")
print(meta_keywords.get("content", "") if meta_keywords else "Not found")

BeautifulSoup supports CSS selectors too. For example, soup.select_one('meta[property="og:title"]') locates an Open Graph title tag. See BeautifulSoup select() for additional selector examples.

Read titles and keyword metadata from video webpages 🔝

You can use BeautifulSoup to inspect metadata present in a video webpage's HTML. The original example uses a public YouTube link, but such external pages change over time, so fixed positions like find_all('meta')[4] are unreliable.

 Copy
import requests
from bs4 import BeautifulSoup

link = "https://youtu.be/SPw6SfN9beg"
response = requests.get(link, timeout=15)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")

page_title = soup.find("title")
keyword_tag = soup.find("meta", attrs={"name": "keywords"})
print(page_title.get_text(strip=True) if page_title else "No title")
print(keyword_tag.get("content", "") if keyword_tag else "No keywords tag")

This reads only metadata present in the fetched HTML. A missing keywords tag is normal on many pages; it does not mean the video has no tags. For large-scale video metadata retrieval, prefer a supported API over scraping changing HTML layouts.

Get all links from a webpage 🔝

Collect anchor tags with find_all("a") and read the href attribute. A webpage can contain relative links, fragments, and links to external websites.

 Copy
import requests
link="https://www.plus2net.com"
content = requests.get(link)
from bs4 import BeautifulSoup
soup = BeautifulSoup(content.text, 'html.parser')

my_links = []
for ln in soup.findAll('a'):
    my_links.append(ln.get('href'))
print(my_links)

Use the current find_all() name and normalize relative URLs with urljoin. This version also skips anchors without an href:

 Copy
from urllib.parse import urljoin
from bs4 import BeautifulSoup
import requests

url = "https://www.plus2net.com/"
response = requests.get(url, timeout=15)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")

links = []
for tag in soup.find_all("a", href=True):
    absolute_url = urljoin(response.url, tag["href"])
    if absolute_url.startswith(("https://", "https://")):
        links.append(absolute_url)

for link in sorted(set(links)):
    print(link)

You can then save the unique links to a file or compare them with your site's sitemap. If you need to audit SEO elements from multiple pages, see the BeautifulSoup webpage auditor.

Check canonical URL matching using MySQL 🔝

This example checks whether a page's HTML contains the expected canonical URL stored in a MySQL table. It combines Requests, BeautifulSoup, and SQLAlchemy. The database table in this example is java_tutorial with columns url and f_name; change them to match your own setup.

Install the extra database libraries if you plan to run this section:

pip install sqlalchemy mysqlclient

 Copy
import requests
from bs4 import BeautifulSoup
from sqlalchemy import create_engine, text

# Replace the example database credentials with your own settings.
engine = create_engine("mysql+mysqldb://id:pw@localhost/my_tutorial")
query = text("SELECT url, f_name FROM java_tutorial WHERE tag_canonical=1 AND tag_head=1")

with engine.connect() as connection:
    rows = connection.execute(query).mappings().all()

for row in rows:
    local_url = "http://localhost/plus2net/java_tutorial/" + row["f_name"]
    try:
        response = requests.get(local_url, timeout=15)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "html.parser")
        tag = soup.find("link", rel="canonical")
        actual = tag.get("href", "").strip() if tag else ""
        expected = row["url"].strip()
        if actual != expected:
            print("Mismatch:", local_url, "Expected:", expected, "Found:", actual)
    except requests.RequestException as err:
        print("Fetch failed:", local_url, err)

The script prints only pages whose canonical link differs from the expected URL. Missing canonical tags produce an empty actual value. Verify URLs after redirects, query strings, and the required canonical policy for your site before deciding whether a difference is an error.

Canonical audit workflow

1. Read the expected production URL and filename from the database.
2. Fetch the local or staging HTML page.
3. Find the single <link rel="canonical"> element.
4. Compare its href with the expected production URL and report mismatches.

When HTML is already stored in a database, you can parse that stored content directly rather than requesting every page again. For a more complete audit workflow, see Python webpage auditor with SQLite and MySQL analysis.

Generate a dynamic table of contents with BeautifulSoup 🔝

The code below builds unique heading IDs and corresponding links, then copies the generated markup to the clipboard with pyperclip.

pip install pyperclip

 Copy
import re
import requests
import pyperclip
from bs4 import BeautifulSoup

url = "http://localhost/plus2net/php_tutorial/sqlite-paging.php"
response = requests.get(url, timeout=15)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")

headings = soup.find_all("h2")
links = []
used_ids = set()
for index, heading in enumerate(headings, start=1):
    title = heading.get_text(" ", strip=True)
    base = re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-")
    base = base or "section"
    anchor = base
    suffix = 2
    while anchor in used_ids:
        anchor = f"{base}-{suffix}"
        suffix += 1
    used_ids.add(anchor)
    heading["id"] = anchor
    item = soup.new_tag("li")
    link = soup.new_tag("a", href="#" + anchor)
    link.string = title
    item.append(link)
    links.append(item)

nav = soup.new_tag("nav")
nav["aria-label"] = "Table of contents"
ul = soup.new_tag("ul")
for item in links:
    ul.append(item)
nav.append(ul)

result = str(nav)
pyperclip.copy(result)
print(result)

# Save str(soup) if you also want the generated heading IDs in the HTML.

Important: A table of contents works only when its target headings have the corresponding id attributes. In the example, soup contains those updated IDs, while result contains only the navigation HTML.

Common errors and troubleshooting 🔝

Responsible scraping: Review the site's terms and robots guidance, honor access controls, avoid sending excessive requests, and do not collect private information without authorization.

Frequently asked questions 🔝

What is the difference between Requests and BeautifulSoup?
Requests downloads HTTP responses. BeautifulSoup parses HTML or XML that you already have; it does not fetch webpages on its own.

When should I use find(), find_all(), or select()?
find() returns one matching element, find_all() returns a list of matches, and select() uses CSS selectors.

How do I extract only visible text?
Use element.get_text(" ", strip=True) after selecting the relevant part of the page. Remove scripts and styling elements first when appropriate.

Can BeautifulSoup read XML?
Yes. Install lxml and use BeautifulSoup(xml_content, "xml") for XML-specific parsing.

Why is a heading's .string sometimes None?
An element with nested text nodes or child tags might not have a single direct string. Use get_text() when you want combined text.

Related BeautifulSoup tutorials 🔝

Continue with focused examples in the same Python learning sequence:

• BeautifulSoup find_all() to collect matching tags
• BeautifulSoup children and parent elements
• Select HTML elements with CSS selectors
• Build a Python webpage auditor for SEO analysis
• Check GA4 analytics tags on webpages
• Collect search suggestions using Python
• Automate mutual fund NAV tracking using requests

Learn to store file information in a MySQL table if you want to extend a scraper into a database audit.

Subhendu Mohapatra

Author & Instructor at plus2net

I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.

About the author   Watch on YouTube

Connect: