Web Scraping with Python and BeautifulSoup: A Practical Beginner's Guide

The web is the largest data source in the world, but most of that data is locked inside HTML pages designed for humans, not programs. Web scraping is the technique of fetching those pages and pulling out the pieces you care about: product prices, article headlines, job listings, research data, and more.

In this guide, you'll learn how to scrape websites with Python and BeautifulSoup, one of the friendliest parsing libraries around. By the end, you'll have built a complete scraper that walks through multiple pages and saves its results to a CSV file.

What Is Web Scraping?

Web scraping is the automated process of requesting a web page, reading its HTML, and extracting structured data from it. A scraper typically does four things:

1. Fetches a page over HTTP.

2. Parses the HTML into a searchable structure.

3. Extracts the elements it needs (text, links, attributes).

4. Stores the results in a useful format such as CSV, JSON, or a database.

Python is a popular choice for this because of its readable syntax and an excellent ecosystem of libraries.

Why BeautifulSoup?

BeautifulSoup (package name beautifulsoup4) is a library for parsing HTML and XML. It doesn't download pages itself; it focuses on making the parsed document easy to navigate and search. Here's why it's a great starting point:

• Forgiving: it handles messy, broken HTML that real websites are full of.

• Intuitive: searching by tag, class, id, or CSS selector reads almost like English.

• Lightweight: no browser, no heavy setup.

It pairs naturally with the requests library, which handles the HTTP side.

Before You Scrape: Be a Good Citizen

Scraping is powerful, so use it responsibly:

• Read the site's Terms of Service. Some sites prohibit scraping.

• Check robots.txt. Found at https://example.com/robots.txt, it tells crawlers which paths are off limits.

• Prefer official APIs when they exist. They're more stable and often explicitly permitted.

• Don't hammer servers. Add delays between requests.

• Identify yourself with a descriptive User-Agent header.

• Avoid personal or sensitive data, and respect copyright and privacy laws that apply to you.

Python can even check robots.txt for you:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

print(rp.can_fetch("MyLearningScraper", "https://example.com/some/page"))

Setting Up

You'll need Python 3.9 or newer (the full scraper below uses built-in generic type hints like list[dict]). Create a virtual environment and install the two libraries:

python -m venv venv
source venv/bin/activate        # On Windows: venv\Scripts\activate
pip install requests beautifulsoup4

Your First Scrape

Let's fetch a page and read its title. We'll use books.toscrape.com, a sandbox site built specifically for practicing scraping.

import requests
from bs4 import BeautifulSoup

url = "https://books.toscrape.com/"
response = requests.get(url, timeout=10)
response.raise_for_status()          # Raise an error for 4xx/5xx responses

soup = BeautifulSoup(response.content, "html.parser")

print(soup.title.get_text(strip=True))

A few notes on what happened:

• requests.get() downloads the page. Always set a timeout so your script can't hang forever.

• raise_for_status() turns bad HTTP responses into exceptions you can handle.

• We pass response.content (raw bytes) so BeautifulSoup can detect the encoding from the page itself. Some servers don't declare a charset in their headers, and in that case requests falls back to ISO-8859-1 for response.text, which garbles characters like £ into £.

• "html.parser" is Python's built-in parser. You can also install lxml for extra speed.

Understanding the Soup

Once parsed, the document is a tree of objects you can navigate. Here's a tiny example using an HTML string so you can see the basics without any network calls:

from bs4 import BeautifulSoup

html = """
<div class="product" id="p1">
  <h2>Mechanical Keyboard</h2>
  <span class="price">$89.99</span>
  <a href="/products/keyboard">View details</a>
</div>
<div class="product" id="p2">
  <h2>Wireless Mouse</h2>
  <span class="price">$24.50</span>
  <a href="/products/mouse">View details</a>
</div>
"""

soup = BeautifulSoup(html, "html.parser")

Finding elements

# First matching element
first = soup.find("div", class_="product")
print(first.h2.get_text(strip=True))        # Mechanical Keyboard

# All matching elements
for product in soup.find_all("div", class_="product"):
    name = product.find("h2").get_text(strip=True)
    price = product.find("span", class_="price").get_text(strip=True)
    print(name, price)

Note: class is a reserved word in Python, so BeautifulSoup uses class_ (with an underscore).

Using CSS selectors

If you know CSS, select() and select_one() are often cleaner:

# All prices
prices = [el.get_text(strip=True) for el in soup.select("div.product span.price")]
print(prices)                                # ['$89.99', '$24.50']

# One element by id
mouse = soup.select_one("#p2 h2")
print(mouse.get_text(strip=True))            # Wireless Mouse

Reading attributes

Elements behave like dictionaries for their attributes:

link = soup.select_one("a")
print(link["href"])                          # /products/keyboard
print(link.get("href"))                      # Same, but returns None if missing

Use .get() when an attribute might not exist, which prevents a KeyError.

Quick reference

Building a Real Scraper

Now let's put it all together. Our goal: scrape every book on books.toscrape.com, collecting the title, price, rating, availability, and URL, then save everything to a CSV.

First, inspect the page using your browser's developer tools (right-click, then Inspect). You'll find each book lives inside an <article class="product_pod">, with the title in the link inside <h3>, the price in p.price_color, and the rating encoded as a class name like star-rating Three.

import csv
import re
import time
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

BASE_URL = "https://books.toscrape.com/"
HEADERS = {"User-Agent": "MyLearningScraper/1.0 (educational project)"}
RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}


def fetch(url: str) -> BeautifulSoup:
    """Download a page and return it as a BeautifulSoup object."""
    response = requests.get(url, headers=HEADERS, timeout=10)
    response.raise_for_status()
    return BeautifulSoup(response.content, "html.parser")


def parse_book(card, page_url: str) -> dict:
    """Extract the fields we care about from one book card."""
    link = card.h3.a
    rating_word = card.select_one("p.star-rating")["class"][1]
    price_text = card.select_one("p.price_color").get_text(strip=True)
    availability = card.select_one("p.availability").get_text(strip=True)

    return {
        "title": link["title"],
        "price": float(re.sub(r"[^\d.]", "", price_text)),
        "rating": RATINGS.get(rating_word),
        "in_stock": "In stock" in availability,
        "url": urljoin(page_url, link["href"]),
    }


def scrape_all(start_url: str, max_pages: int = 50, delay: float = 1.0) -> list[dict]:
    """Follow the 'next' links page by page, collecting every book."""
    books = []
    url = start_url
    page = 0

    while url and page < max_pages:
        page += 1
        print(f"Scraping page {page}: {url}")

        soup = fetch(url)
        for card in soup.select("article.product_pod"):
            books.append(parse_book(card, url))

        next_link = soup.select_one("li.next a")
        url = urljoin(url, next_link["href"]) if next_link else None

        time.sleep(delay)            # Be polite: pause between requests

    return books


def save_to_csv(rows: list[dict], filename: str) -> None:
    """Write the scraped rows to a CSV file."""
    if not rows:
        print("Nothing to save.")
        return

    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=rows[0].keys())
        writer.writeheader()
        writer.writerows(rows)

    print(f"Saved {len(rows)} rows to {filename}")


if __name__ == "__main__":
    books = scrape_all(BASE_URL)
    save_to_csv(books, "books.csv")

Save it as scraper.py and run python scraper.py. The site has 50 pages of 20 books each, and with a one-second pause between requests the crawl takes a minute or two, leaving you with a books.csv of about 1,000 books.

What's worth noticing

• Small, focused functions. Fetching, parsing, crawling, and saving are separate, so each is easy to test and change.

• urljoin for links. Pages often use relative links like catalogue/page-2.html. urljoin resolves them against the current URL correctly.

• Pagination by following "next". Rather than guessing page numbers, the scraper follows the site's own li.next a link and stops when it disappears.

• Politeness built in. A custom User-Agent, a timeout, and a delay between requests.

Making Your Scraper More Robust

Real-world scraping is messy. Here are the issues you'll most likely run into.

Handle network errors and retries

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retries = Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504])
session.mount("https://", HTTPAdapter(max_retries=retries))
session.headers.update({"User-Agent": "MyLearningScraper/1.0"})

response = session.get("https://books.toscrape.com/", timeout=10)

A Session reuses connections (faster) and shares headers across requests. The retry adapter automatically retries temporary failures with increasing delays.

Guard against missing elements

Pages are inconsistent. If select_one() finds nothing, it returns None, and calling .get_text() on None crashes your script. Defend against it:

price_el = card.select_one("p.price_color")
price = price_el.get_text(strip=True) if price_el else None

Watch for site changes

Scrapers break when a site's HTML changes. Prefer stable hooks (ids, semantic classes, data attributes) over fragile ones (deeply nested positions like "the third div inside the second div"), and log or alert when a page doesn't match your expectations.

When BeautifulSoup Isn't Enough

BeautifulSoup only sees the HTML the server sends. Many modern sites load content with JavaScript after the page opens, so the data you see in your browser isn't in the raw HTML. In that case you have a few options:

1. Look for a hidden API. Open the Network tab in your browser's developer tools; the page may fetch JSON from an endpoint you can call directly with requests. This is often the cleanest solution.

2. Use a browser automation tool like Playwright or Selenium to render the page, then hand the final HTML to BeautifulSoup.

3. Use a full crawling framework like Scrapy when you need to crawl large sites with concurrency, pipelines, and built-in throttling.

Wrapping Up

You've now seen the complete scraping workflow: fetch a page with requests, parse it with BeautifulSoup, locate elements with find/select, extract text and attributes, follow pagination, and save the results. The same pattern scales from a ten-line script to a serious data pipeline.

A few next steps to practice:

• Scrape another sandbox, such as quotes.toscrape.com, and try extracting authors and tags.

• Save your data as JSON or load it into a pandas DataFrame for analysis.

• Add logging and retries to make your scraper production-ready.

Happy scraping, and remember to scrape responsibly!

Found this helpful? Share it with a fellow developer or drop your questions in the comments.