--- title: "Web Scraping with Python and BeautifulSoup: A Practical Beginner's Guide" author: "Bhavya Arora" published: "2026-11-12T06:00:06.287Z" updated: "2026-11-12T06:00:06.287Z" url: "https://zyvop.com/web-scraping-with-python-and-beautifulsoup-a-practical-beginner-s-guide-rivrw" canonical: "https://zyvop.com/web-scraping-with-python-and-beautifulsoup-a-practical-beginner-s-guide-rivrw" tags: ["Web Scraping","data","BeautifulSoup","Python"] --- # Web Scraping with Python and BeautifulSoup: A Practical Beginner's Guide A beginner-friendly walkthrough of web scraping in Python: fetch pages, parse HTML with BeautifulSoup, and export your results to CSV. ![Cover image]() The web is the largest data source in the world, but most of that data is locked inside HTML pages designed for humans, not programs. **Web scraping** is the technique of fetching those pages and pulling out the pieces you care about: product prices, article headlines, job listings, research data, and more. In this guide, you'll learn how to scrape websites with **Python** and **BeautifulSoup**, one of the friendliest parsing libraries around. By the end, you'll have built a complete scraper that walks through multiple pages and saves its results to a CSV file. * * * ## What Is Web Scraping? Web scraping is the automated process of requesting a web page, reading its HTML, and extracting structured data from it. A scraper typically does four things: 1. **Fetches** a page over HTTP. 2. **Parses** the HTML into a searchable structure. 3. **Extracts** the elements it needs (text, links, attributes). 4. **Stores** the results in a useful format such as CSV, JSON, or a database. Python is a popular choice for this because of its readable syntax and an excellent ecosystem of libraries. ## Why BeautifulSoup? [BeautifulSoup]() (package name `beautifulsoup4`) is a library for parsing HTML and XML. It doesn't download pages itself; it focuses on making the parsed document easy to navigate and search. Here's why it's a great starting point: - **Forgiving:** it handles messy, broken HTML that real websites are full of. - **Intuitive:** searching by tag, class, id, or CSS selector reads almost like English. - **Lightweight:** no browser, no heavy setup. It pairs naturally with the `requests` library, which handles the HTTP side. ## Before You Scrape: Be a Good Citizen Scraping is powerful, so use it responsibly: - **Read the site's Terms of Service.** Some sites prohibit scraping. - **Check** `robots.txt`**.** Found at `https://example.com/robots.txt`, it tells crawlers which paths are off limits. - **Prefer official APIs** when they exist. They're more stable and often explicitly permitted. - **Don't hammer servers.** Add delays between requests. - **Identify yourself** with a descriptive `User-Agent` header. - **Avoid personal or sensitive data,** and respect copyright and privacy laws that apply to you. Python can even check `robots.txt` for you: ```python from urllib.robotparser import RobotFileParser rp = RobotFileParser() rp.set_url("https://example.com/robots.txt") rp.read() print(rp.can_fetch("MyLearningScraper", "https://example.com/some/page")) ``` ## Setting Up You'll need **Python 3.9 or newer** (the full scraper below uses built-in generic type hints like `list[dict]`). Create a virtual environment and install the two libraries: ```bash python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate pip install requests beautifulsoup4 ``` ## Your First Scrape Let's fetch a page and read its title. We'll use [books.toscrape.com](), a sandbox site built specifically for practicing scraping. ```python import requests from bs4 import BeautifulSoup url = "https://books.toscrape.com/" response = requests.get(url, timeout=10) response.raise_for_status() # Raise an error for 4xx/5xx responses soup = BeautifulSoup(response.content, "html.parser") print(soup.title.get_text(strip=True)) ``` A few notes on what happened: - `requests.get()` downloads the page. Always set a `timeout` so your script can't hang forever. - `raise_for_status()` turns bad HTTP responses into exceptions you can handle. - We pass `response.content` (raw bytes) so BeautifulSoup can detect the encoding from the page itself. Some servers don't declare a charset in their headers, and in that case `requests` falls back to `ISO-8859-1` for `response.text`, which garbles characters like `£` into `£`. - `"html.parser"` is Python's built-in parser. You can also install `lxml` for extra speed. ## Understanding the Soup Once parsed, the document is a tree of objects you can navigate. Here's a tiny example using an HTML string so you can see the basics without any network calls: ```python from bs4 import BeautifulSoup html = """

Mechanical Keyboard

$89.99 View details

Wireless Mouse

$24.50 View details
""" soup = BeautifulSoup(html, "html.parser") ``` ### Finding elements ```python # First matching element first = soup.find("div", class_="product") print(first.h2.get_text(strip=True)) # Mechanical Keyboard # All matching elements for product in soup.find_all("div", class_="product"): name = product.find("h2").get_text(strip=True) price = product.find("span", class_="price").get_text(strip=True) print(name, price) ``` > **Note:** `class` is a reserved word in Python, so BeautifulSoup uses `class_` (with an underscore). ### Using CSS selectors If you know CSS, `select()` and `select_one()` are often cleaner: ```python # All prices prices = [el.get_text(strip=True) for el in soup.select("div.product span.price")] print(prices) # ['$89.99', '$24.50'] # One element by id mouse = soup.select_one("#p2 h2") print(mouse.get_text(strip=True)) # Wireless Mouse ``` ### Reading attributes Elements behave like dictionaries for their attributes: ```python link = soup.select_one("a") print(link["href"]) # /products/keyboard print(link.get("href")) # Same, but returns None if missing ``` Use `.get()` when an attribute might not exist, which prevents a `KeyError`. ### Quick reference | Task | Code | | --- | --- | | First match by tag/class | `soup.find("div", class_="x")` | | All matches | `soup.find_all("a")` | | CSS selector (one) | `soup.select_one("ul.menu > li")` | | CSS selector (many) | `soup.select("a[href]")` | | Text of an element | `el.get_text(strip=True)` | | Attribute value | `el["href"]` or `el.get("href")` | | Parent / siblings | `el.parent`, `el.find_next_sibling()` | ## Building a Real Scraper Now let's put it all together. Our goal: scrape every book on books.toscrape.com, collecting the **title, price, rating, availability, and URL**, then save everything to a CSV. First, inspect the page using your browser's developer tools (right-click, then **Inspect**). You'll find each book lives inside an `
`, with the title in the link inside `

`, the price in `p.price_color`, and the rating encoded as a class name like `star-rating Three`. ```python import csv import re import time from urllib.parse import urljoin import requests from bs4 import BeautifulSoup BASE_URL = "https://books.toscrape.com/" HEADERS = {"User-Agent": "MyLearningScraper/1.0 (educational project)"} RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5} def fetch(url: str) -> BeautifulSoup: """Download a page and return it as a BeautifulSoup object.""" response = requests.get(url, headers=HEADERS, timeout=10) response.raise_for_status() return BeautifulSoup(response.content, "html.parser") def parse_book(card, page_url: str) -> dict: """Extract the fields we care about from one book card.""" link = card.h3.a rating_word = card.select_one("p.star-rating")["class"][1] price_text = card.select_one("p.price_color").get_text(strip=True) availability = card.select_one("p.availability").get_text(strip=True) return { "title": link["title"], "price": float(re.sub(r"[^\d.]", "", price_text)), "rating": RATINGS.get(rating_word), "in_stock": "In stock" in availability, "url": urljoin(page_url, link["href"]), } def scrape_all(start_url: str, max_pages: int = 50, delay: float = 1.0) -> list[dict]: """Follow the 'next' links page by page, collecting every book.""" books = [] url = start_url page = 0 while url and page < max_pages: page += 1 print(f"Scraping page {page}: {url}") soup = fetch(url) for card in soup.select("article.product_pod"): books.append(parse_book(card, url)) next_link = soup.select_one("li.next a") url = urljoin(url, next_link["href"]) if next_link else None time.sleep(delay) # Be polite: pause between requests return books def save_to_csv(rows: list[dict], filename: str) -> None: """Write the scraped rows to a CSV file.""" if not rows: print("Nothing to save.") return with open(filename, "w", newline="", encoding="utf-8") as f: writer = csv.DictWriter(f, fieldnames=rows[0].keys()) writer.writeheader() writer.writerows(rows) print(f"Saved {len(rows)} rows to {filename}") if __name__ == "__main__": books = scrape_all(BASE_URL) save_to_csv(books, "books.csv") ``` Save it as `scraper.py` and run `python scraper.py`. The site has 50 pages of 20 books each, and with a one-second pause between requests the crawl takes a minute or two, leaving you with a `books.csv` of about 1,000 books. ### What's worth noticing - **Small, focused functions.** Fetching, parsing, crawling, and saving are separate, so each is easy to test and change. - `urljoin` **for links.** Pages often use relative links like `catalogue/page-2.html`. `urljoin` resolves them against the current URL correctly. - **Pagination by following "next".** Rather than guessing page numbers, the scraper follows the site's own `li.next a` link and stops when it disappears. - **Politeness built in.** A custom `User-Agent`, a timeout, and a delay between requests. ## Making Your Scraper More Robust Real-world scraping is messy. Here are the issues you'll most likely run into. ### Handle network errors and retries ```python import requests from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry session = requests.Session() retries = Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504]) session.mount("https://", HTTPAdapter(max_retries=retries)) session.headers.update({"User-Agent": "MyLearningScraper/1.0"}) response = session.get("https://books.toscrape.com/", timeout=10) ``` A `Session` reuses connections (faster) and shares headers across requests. The retry adapter automatically retries temporary failures with increasing delays. ### Guard against missing elements Pages are inconsistent. If `select_one()` finds nothing, it returns `None`, and calling `.get_text()` on `None` crashes your script. Defend against it: ```python price_el = card.select_one("p.price_color") price = price_el.get_text(strip=True) if price_el else None ``` ### Watch for site changes Scrapers break when a site's HTML changes. Prefer stable hooks (ids, semantic classes, data attributes) over fragile ones (deeply nested positions like "the third `div` inside the second `div`"), and log or alert when a page doesn't match your expectations. ## When BeautifulSoup Isn't Enough BeautifulSoup only sees the HTML the server sends. Many modern sites load content with JavaScript after the page opens, so the data you see in your browser isn't in the raw HTML. In that case you have a few options: 1. **Look for a hidden API.** Open the Network tab in your browser's developer tools; the page may fetch JSON from an endpoint you can call directly with `requests`. This is often the cleanest solution. 2. **Use a browser automation tool** like [Playwright]() or Selenium to render the page, then hand the final HTML to BeautifulSoup. 3. **Use a full crawling framework** like [Scrapy]() when you need to crawl large sites with concurrency, pipelines, and built-in throttling. ## Wrapping Up You've now seen the complete scraping workflow: fetch a page with `requests`, parse it with BeautifulSoup, locate elements with `find`/`select`, extract text and attributes, follow pagination, and save the results. The same pattern scales from a ten-line script to a serious data pipeline. A few next steps to practice: - Scrape another sandbox, such as [quotes.toscrape.com](), and try extracting authors and tags. - Save your data as JSON or load it into a pandas DataFrame for analysis. - Add logging and retries to make your scraper production-ready. Happy scraping, and remember to scrape responsibly! * * * _Found this helpful? Share it with a fellow developer or drop your questions in the comments._