{"schemaVersion":"1.0","type":"TechArticle","types":["Article","TechArticle"],"slug":"web-scraping-with-python-and-beautifulsoup-a-practical-beginner-s-guide-rivrw","url":"https://zyvop.com/web-scraping-with-python-and-beautifulsoup-a-practical-beginner-s-guide-rivrw","title":"Web Scraping with Python and BeautifulSoup: A Practical Beginner's Guide","subtitle":"A beginner-friendly walkthrough of web scraping in Python: fetch pages, parse HTML with BeautifulSoup, and export your results to CSV.","tldr":"The web is the largest data source in the world, but most of that data is locked inside HTML pages designed for humans, not programs.","keywords":["Web Scraping","data","BeautifulSoup","Python","Tutorial"],"entities":["Bhavya Arora","Web Scraping","data","BeautifulSoup","Python","Tutorial","ZyVOP"],"keyTakeaways":["The web is the largest data source in the world, but most of that data is locked inside HTML pages designed for humans, not programs.","Web scraping is the technique of fetching those pages and pulling out the pieces you care about: product prices, article headlines, job listings, research data, and more.","In this guide, you'll learn how to scrape websites with Python and BeautifulSoup, one of the friendliest parsing libraries around."],"headings":["What Is Web Scraping?","Why BeautifulSoup?","Before You Scrape: Be a Good Citizen","Setting Up","Your First Scrape","Understanding the Soup","Finding elements","Using CSS selectors","Reading attributes","Quick reference","Building a Real Scraper","What's worth noticing","Making Your Scraper More Robust","Handle network errors and retries","Guard against missing elements","Watch for site changes","When BeautifulSoup Isn't Enough","Wrapping Up"],"outboundLinks":["https://www.crummy.com/software/BeautifulSoup/","https://books.toscrape.com/","https://playwright.dev/python/","https://scrapy.org/","https://quotes.toscrape.com/"],"contentText":"The web is the largest data source in the world, but most of that data is locked inside HTML pages designed for humans, not programs. Web scraping is the technique of fetching those pages and pulling out the pieces you care about: product prices, article headlines, job listings, research data, and more. In this guide, you'll learn how to scrape websites with Python and BeautifulSoup, one of the friendliest parsing libraries around. By the end, you'll have built a complete scraper that walks through multiple pages and saves its results to a CSV file. What Is Web Scraping? Web scraping is the automated process of requesting a web page, reading its HTML, and extracting structured data from it. A scraper typically does four things: Fetches a page over HTTP. Parses the HTML into a searchable structure. Extracts the elements it needs (text, links, attributes). Stores the results in a useful format such as CSV, JSON, or a database. Python is a popular choice for this because of its readable syntax and an excellent ecosystem of libraries. Why BeautifulSoup? BeautifulSoup (package name beautifulsoup4) is a library for parsing HTML and XML. It doesn't download pages itself; it focuses on making the parsed document easy to navigate and search. Here's why it's a great starting point: Forgiving: it handles messy, broken HTML that real websites are full of. Intuitive: searching by tag, class, id, or CSS selector reads almost like English. Lightweight: no browser, no heavy setup. It pairs naturally with the requests library, which handles the HTTP side. Before You Scrape: Be a Good Citizen Scraping is powerful, so use it responsibly: Read the site's Terms of Service. Some sites prohibit scraping. Check robots.txt. Found at https://example.com/robots.txt, it tells crawlers which paths are off limits. Prefer official APIs when they exist. They're more stable and often explicitly permitted. Don't hammer servers. Add delays between requests. Identify yourself with a descriptive User-Agent header. Avoid personal or sensitive data, and respect copyright and privacy laws that apply to you. Python can even check robots.txt for you: from urllib.robotparser import RobotFileParser rp = RobotFileParser() rp.set_url(\"https://example.com/robots.txt\") rp.read() print(rp.can_fetch(\"MyLearningScraper\", \"https://example.com/some/page\"))Setting Up You'll need Python 3.9 or newer (the full scraper below uses built-in generic type hints like list[dict]). Create a virtual environment and install the two libraries: python -m venv venv source venv/bin/activate # On Windows: venv\\Scripts\\activate pip install requests beautifulsoup4Your First Scrape Let's fetch a page and read its title. We'll use books.toscrape.com, a sandbox site built specifically for practicing scraping. import requests from bs4 import BeautifulSoup url = \"https://books.toscrape.com/\" response = requests.get(url, timeout=10) response.raise_for_status() # Raise an error for 4xx/5xx responses soup = BeautifulSoup(response.content, \"html.parser\") print(soup.title.get_text(strip=True))A few notes on what happened: requests.get() downloads the page. Always set a timeout so your script can't hang forever. raise_for_status() turns bad HTTP responses into exceptions you can handle. We pass response.content (raw bytes) so BeautifulSoup can detect the encoding from the page itself. Some servers don't declare a charset in their headers, and in that case requests falls back to ISO-8859-1 for response.text, which garbles characters like £ into £. \"html.parser\" is Python's built-in parser. You can also install lxml for extra speed. Understanding the Soup Once parsed, the document is a tree of objects you can navigate. Here's a tiny example using an HTML string so you can see the basics without any network calls: from bs4 import BeautifulSoup html = \"\"\" <div class=\"product\" id=\"p1\"> <h2>Mechanical Keyboard</h2> <span class=\"price\">$89.99</span> <a href=\"/products/keyboard\">View details</a> </div> <div class=\"product\" id=\"p2\"> <h2>Wireless Mouse</h2> <span class=\"price\">$24.50</span> <a href=\"/products/mouse\">View details</a> </div> \"\"\" soup = BeautifulSoup(html, \"html.parser\")Finding elements # First matching element first = soup.find(\"div\", class_=\"product\") print(first.h2.get_text(strip=True)) # Mechanical Keyboard # All matching elements for product in soup.find_all(\"div\", class_=\"product\"): name = product.find(\"h2\").get_text(strip=True) price = product.find(\"span\", class_=\"price\").get_text(strip=True) print(name, price)Note: class is a reserved word in Python, so BeautifulSoup uses class_ (with an underscore). Using CSS selectors If you know CSS, select() and select_one() are often cleaner: # All prices prices = [el.get_text(strip=True) for el in soup.select(\"div.product span.price\")] print(prices) # ['$89.99', '$24.50'] # One element by id mouse = soup.select_one(\"#p2 h2\") print(mouse.get_text(strip=True)) # Wireless MouseReading attributes Elements behave like dictionaries for their attributes: link = soup.select_one(\"a\") print(link[\"href\"]) # /products/keyboard print(link.get(\"href\")) # Same, but returns None if missingUse .get() when an attribute might not exist, which prevents a KeyError. Quick reference Task Code First match by tag/class soup.find(\"div\", class_=\"x\") All matches soup.find_all(\"a\") CSS selector (one) soup.select_one(\"ul.menu > li\") CSS selector (many) soup.select(\"a[href]\") Text of an element el.get_text(strip=True) Attribute value el[\"href\"] or el.get(\"href\") Parent / siblings el.parent, el.find_next_sibling() Building a Real Scraper Now let's put it all together. Our goal: scrape every book on books.toscrape.com, collecting the title, price, rating, availability, and URL, then save everything to a CSV. First, inspect the page using your browser's developer tools (right-click, then Inspect). You'll find each book lives inside an <article class=\"product_pod\">, with the title in the link inside <h3>, the price in p.price_color, and the rating encoded as a class name like star-rating Three. import csv import re import time from urllib.parse import urljoin import requests from bs4 import BeautifulSoup BASE_URL = \"https://books.toscrape.com/\" HEADERS = {\"User-Agent\": \"MyLearningScraper/1.0 (educational project)\"} RATINGS = {\"One\": 1, \"Two\": 2, \"Three\": 3, \"Four\": 4, \"Five\": 5} def fetch(url: str) -> BeautifulSoup: \"\"\"Download a page and return it as a BeautifulSoup object.\"\"\" response = requests.get(url, headers=HEADERS, timeout=10) response.raise_for_status() return BeautifulSoup(response.content, \"html.parser\") def parse_book(card, page_url: str) -> dict: \"\"\"Extract the fields we care about from one book card.\"\"\" link = card.h3.a rating_word = card.select_one(\"p.star-rating\")[\"class\"][1] price_text = card.select_one(\"p.price_color\").get_text(strip=True) availability = card.select_one(\"p.availability\").get_text(strip=True) return { \"title\": link[\"title\"], \"price\": float(re.sub(r\"[^\\d.]\", \"\", price_text)), \"rating\": RATINGS.get(rating_word), \"in_stock\": \"In stock\" in availability, \"url\": urljoin(page_url, link[\"href\"]), } def scrape_all(start_url: str, max_pages: int = 50, delay: float = 1.0) -> list[dict]: \"\"\"Follow the 'next' links page by page, collecting every book.\"\"\" books = [] url = start_url page = 0 while url and page < max_pages: page += 1 print(f\"Scraping page {page}: {url}\") soup = fetch(url) for card in soup.select(\"article.product_pod\"): books.append(parse_book(card, url)) next_link = soup.select_one(\"li.next a\") url = urljoin(url, next_link[\"href\"]) if next_link else None time.sleep(delay) # Be polite: pause between requests return books def save_to_csv(rows: list[dict], filename: str) -> None: \"\"\"Write the scraped rows to a CSV file.\"\"\" if not rows: print(\"Nothing to save.\") return with open(filename, \"w\", newline=\"\", encoding=\"utf-8\") as f: writer = csv.DictWriter(f, fieldnames=rows[0].keys()) writer.writeheader() writer.writerows(rows) print(f\"Saved {len(rows)} rows to {filename}\") if __name__ == \"__main__\": books = scrape_all(BASE_URL) save_to_csv(books, \"books.csv\")Save it as scraper.py and run python scraper.py. The site has 50 pages of 20 books each, and with a one-second pause between requests the crawl takes a minute or two, leaving you with a books.csv of about 1,000 books. What's worth noticing Small, focused functions. Fetching, parsing, crawling, and saving are separate, so each is easy to test and change. urljoin for links. Pages often use relative links like catalogue/page-2.html. urljoin resolves them against the current URL correctly. Pagination by following \"next\". Rather than guessing page numbers, the scraper follows the site's own li.next a link and stops when it disappears. Politeness built in. A custom User-Agent, a timeout, and a delay between requests. Making Your Scraper More Robust Real-world scraping is messy. Here are the issues you'll most likely run into. Handle network errors and retries import requests from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry session = requests.Session() retries = Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504]) session.mount(\"https://\", HTTPAdapter(max_retries=retries)) session.headers.update({\"User-Agent\": \"MyLearningScraper/1.0\"}) response = session.get(\"https://books.toscrape.com/\", timeout=10)A Session reuses connections (faster) and shares headers across requests. The retry adapter automatically retries temporary failures with increasing delays. Guard against missing elements Pages are inconsistent. If select_one() finds nothing, it returns None, and calling .get_text() on None crashes your script. Defend against it: price_el = card.select_one(\"p.price_color\") price = price_el.get_text(strip=True) if price_el else NoneWatch for site changes Scrapers break when a site's HTML changes. Prefer stable hooks (ids, semantic classes, data attributes) over fragile ones (deeply nested positions like \"the third div inside the second div\"), and log or alert when a page doesn't match your expectations. When BeautifulSoup Isn't Enough BeautifulSoup only sees the HTML the server sends. Many modern sites load content with JavaScript after the page opens, so the data you see in your browser isn't in the raw HTML. In that case you have a few options: Look for a hidden API. Open the Network tab in your browser's developer tools; the page may fetch JSON from an endpoint you can call directly with requests. This is often the cleanest solution. Use a browser automation tool like Playwright or Selenium to render the page, then hand the final HTML to BeautifulSoup. Use a full crawling framework like Scrapy when you need to crawl large sites with concurrency, pipelines, and built-in throttling. Wrapping Up You've now seen the complete scraping workflow: fetch a page with requests, parse it with BeautifulSoup, locate elements with find/select, extract text and attributes, follow pagination, and save the results. The same pattern scales from a ten-line script to a serious data pipeline. A few next steps to practice: Scrape another sandbox, such as quotes.toscrape.com, and try extracting authors and tags. Save your data as JSON or load it into a pandas DataFrame for analysis. Add logging and retries to make your scraper production-ready. Happy scraping, and remember to scrape responsibly! Found this helpful? Share it with a fellow developer or drop your questions in the comments.","contentHash":"sha256:029289aeaacce5c82d7e780b8dda079d682bd7112817ee9cd6de1778cc10b4ac","authorName":"Bhavya Arora","authorUrl":"https://zyvop.com/author/bhavya","authorSameAs":[],"category":"Tutorial","tags":["Web Scraping","data","BeautifulSoup","Python"],"audience":"Developers, software engineers, and students learning Tutorial","tone":"Instructional, practical, code-first","readingTimeMinutes":7,"wordCount":1730,"faqs":null,"primaryTopic":"Tutorial","publishedAt":"2026-11-12T06:00:06.287Z","updatedAt":"2026-11-12T06:00:06.287Z","canonicalUrl":"https://zyvop.com/web-scraping-with-python-and-beautifulsoup-a-practical-beginner-s-guide-rivrw"}