
A requests and BeautifulSoup script is the right way to start scraping. But as soon as a job grows, you find yourself writing the same plumbing again and again: a loop over pages, retries, delays, de-duplication, saving results. Soon your 30-line script is 300 lines and mostly not about the data.
Scrapy is a framework that supplies that plumbing. You describe what to extract in a small class called a spider, and Scrapy handles requests, concurrency, retries, throttling, robots.txt, logging, and exporting results.
In this guide you'll build a complete Scrapy project from scratch. It will crawl every page of a bookstore, follow each book to its detail page, clean the data in a pipeline, and export it to JSON or CSV, politely.
Version note: this guide targets Scrapy 2.20 (released November 2026), which requires Python 3.11 or newer. Scrapy has changed a lot recently, and many tutorials online are out of date. The biggest example: the old
start_requests()method was removed in Scrapy 2.16, and spiders now useasync def start(). If you copy a spider from an older tutorial and it crawls nothing, that's probably why.
Why Scrapy?
Here's what you get without writing it yourself:
Need | In a script | In Scrapy |
|---|---|---|
Many requests at once | Threads or asyncio by hand | Built-in concurrency |
Retrying failures | Your own loop | Built-in retry middleware |
Being polite |
| Download delay and AutoThrottle |
Obeying | Extra code | One setting, on by default |
Skipping repeated URLs | A | Automatic de-duplication |
Cleaning and saving data | Mixed into the scraping code | Item pipelines and feed exports |
Knowing what happened |
| Logs and crawl statistics |
When not to use it. For a single page or a quick one-off, requests and BeautifulSoup are simpler. For sites that build their content with JavaScript, you need a browser: see our guide to scraping JavaScript-heavy sites with Playwright. Scrapy shines when you need to crawl many pages reliably.
Setting Up
Check that you have Python 3.11 or newer, then create a virtual environment and install Scrapy:
python --version # 3.11 or newer
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install scrapy
scrapy versionNow create a project:
scrapy startproject bookscraper
cd bookscraperYou get this layout:
bookscraper/
├── scrapy.cfg # project marker (run commands from this folder)
└── bookscraper/
├── items.py # the shape of the data you extract
├── middlewares.py # hooks into the request/response flow
├── pipelines.py # clean, validate and store items
├── settings.py # all configuration
└── spiders/ # your spiders live hereTwo things in the generated settings.py are worth noticing right away: ROBOTSTXT_OBEY = True, so Scrapy respects robots.txt by default, and a DOWNLOAD_DELAY of 1 second with one request at a time per domain. New projects start out polite.
Explore the Page with the Scrapy Shell
Before writing any spider, experiment on the live page. The shell downloads a URL and gives you a response object to play with:
scrapy shell "https://books.toscrape.com/"We'll use the sandbox site books.toscrape.com, which exists for practicing scraping. Inside the shell, try:
>>> response.css("article.product_pod h3 a::attr(title)").get()
'A Light in the Attic'
>>> response.css("p.price_color::text").getall()[:3]
['£51.77', '£53.74', '£50.10']
>>> response.xpath("//li[@class='next']/a/@href").get()
'catalogue/page-2.html'A few things to learn from this:
response.css()takes CSS selectors, extended with::text(the text of an element) and::attr(name)(an attribute's value).response.xpath()does the same job with XPath, which is handy for odd cases..get()returns the first match, orNoneif there isn't one..getall()returns a list of every match.
The shell is the fastest way to get selectors right, so use it constantly. Type exit() to leave.
Your First Spider
A spider is a class with a name, a place to start, and a parse method that turns a response into data. You can generate a skeleton with scrapy genspider books books.toscrape.com, but it's short enough to write by hand. Create bookscraper/spiders/books.py:
import scrapy
class BooksSpider(scrapy.Spider):
name = "books"
allowed_domains = ["books.toscrape.com"]
start_urls = ["https://books.toscrape.com/"]
def parse(self, response):
for card in response.css("article.product_pod"):
yield {
"title": card.css("h3 a::attr(title)").get(),
"price": card.css("p.price_color::text").get(),
"url": response.urljoin(card.css("h3 a::attr(href)").get()),
}How it works:
nameis how you run the spider.allowed_domainsstops the spider from wandering off to other sites.start_urlslists the first pages to download. Scrapy requests each one and passes the response toparse.yieldhands each result back to Scrapy. A spider doesn'treturna list: it yields items one at a time, and Scrapy streams them to wherever they're going.
Run it and export to a file:
scrapy crawl books -O books.json-O (capital) overwrites the output file, so you always start fresh. Open books.json and you'll have the first page of books.
Watch out for
-o. In projects created with Scrapy 2.20, the generated settings includeFEED_MODE = "create". If the file already exists, lowercase-othen logs an error (Not writing ... because it already exists) and throws that run's items away, yet the command still exits normally, so it's easy to miss. In older setups, and outside a project,-oappends, and appending a second run to a JSON file leaves you with a corrupted file. Use-Owhile you're developing, or pick the behavior explicitly with theFEED_MODEsetting.
If you just want to try a spider quickly without creating a project, scrapy runspider books.py -O books.json runs a single file.
About start_urls and start()
start_urls is the simplest way to say where to begin. When you need more control, such as generating a list of URLs or adding headers, write an async def start() method instead:
class BooksSpider(scrapy.Spider):
name = "books"
allowed_domains = ["books.toscrape.com"]
async def start(self):
for page in range(1, 4):
url = f"https://books.toscrape.com/catalogue/page-{page}.html"
yield scrapy.Request(url, callback=self.parse)This replaces start_requests() from older tutorials. Note that async def and yield together make it an asynchronous generator, which Scrapy knows how to run. A leftover start_requests() isn't an error: the spider simply starts, reports Crawled 0 pages, and finishes without a word of complaint.
Following Pagination
Scraping one page isn't much of a crawler. Add this to the end of parse to follow the "Next" link:
next_page = response.css("li.next a::attr(href)").get()
if next_page:
yield response.follow(next_page, callback=self.parse)response.follow() creates a new request for the link and sends its response back to parse. It handles relative URLs for you (no urljoin needed), and Scrapy automatically skips any URL it has already requested. When there's no Next link on the last page, get() returns None and the crawl ends.
Following Links to Detail Pages
The listing doesn't have everything: the description and product code live on each book's own page. So the spider needs two callbacks: one for listing pages and one for book pages. It will also pass the star rating (which we only read from the listing) along to the second callback.
First, define the shape of your data in bookscraper/items.py. Recent Scrapy templates use a plain dataclass:
from dataclasses import dataclass
@dataclass
class Book:
title: str | None = None
price: str | float | None = None
rating: str | int | None = None
availability: str | None = None
description: str | None = None
upc: str | None = None
url: str | None = NoneNow rewrite the spider:
import scrapy
from bookscraper.items import Book
class BooksSpider(scrapy.Spider):
name = "books"
allowed_domains = ["books.toscrape.com"]
start_urls = ["https://books.toscrape.com/"]
def parse(self, response):
"""Listing page: follow every book, then the Next link."""
for card in response.css("article.product_pod"):
rating = card.css("p.star-rating::attr(class)").get().split()[-1] # "star-rating Three" -> "Three"
yield response.follow(
card.css("h3 a::attr(href)").get(),
callback=self.parse_book,
cb_kwargs={"rating": rating},
)
next_page = response.css("li.next a::attr(href)").get()
if next_page:
yield response.follow(next_page, callback=self.parse)
def parse_book(self, response, rating):
"""Book page: build the item."""
availability = "".join(response.css("p.availability::text").getall())
yield Book(
title=response.css("h1::text").get(),
price=response.css("p.price_color::text").get(),
rating=rating,
availability=" ".join(availability.split()),
description=response.css("#product_description ~ p::text").get(),
upc=response.css("table tr:first-child td::text").get(),
url=response.url,
)Notes on what's new:
Callbacks chain the crawl.
parseyields requests for book pages, and each response goes toparse_book, which yields items. A callback can yield both requests and items.cb_kwargspasses extra data to the next callback as keyword arguments. That's howratingtravels from the listing page to the book page.Whitespace cleanup. The availability text has line breaks and padding around it, so we collapse it with
" ".join(text.split()).Missing data. Some pages will lack a field, a description for instance.
.get()returnsNoneinstead of crashing, and the dataclass defaults toNone.
Cleaning Data with Pipelines
Right now price is a string like £51.77 and rating is a word like Three. Item pipelines are the right place to clean and validate items, keeping that logic out of the spider. Each pipeline is a class with a process_item method. Open bookscraper/pipelines.py:
import re
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
class ValidateAndCleanPipeline:
def process_item(self, item):
adapter = ItemAdapter(item)
if not adapter.get("title") or not adapter.get("price"):
raise DropItem(f"Missing title or price: {adapter.get('url')}")
adapter["price"] = float(re.sub(r"[^\d.]", "", adapter["price"])) # "£51.77" -> 51.77
adapter["rating"] = RATINGS.get(adapter["rating"]) # "Three" -> 3
return item
class DuplicatesPipeline:
def __init__(self):
self.seen = set()
def process_item(self, item):
upc = ItemAdapter(item)["upc"]
if upc in self.seen:
raise DropItem(f"Duplicate book: {upc}")
self.seen.add(upc)
return itemKey ideas:
ItemAdapterlets one pipeline work with dictionaries, dataclasses, orscrapy.Itemobjects through the same interface.Return the item to pass it along, or raise
DropItemto discard it. Dropped items are logged and counted but never exported.The first pipeline drops incomplete items and converts the price and rating to numbers. The second drops repeats using the UPC, a unique product code, because the same book can appear under more than one URL.
Older tutorials write
process_item(self, item, spider). That still runs in 2.20 but prints a deprecation warning, so drop thespiderargument in new code.
Pipelines are off until you switch them on in settings.py. The numbers set the order, and lower runs first:
ITEM_PIPELINES = {
"bookscraper.pipelines.ValidateAndCleanPipeline": 300,
"bookscraper.pipelines.DuplicatesPipeline": 400,
}Crawling Politely: Settings
Scrapy is fast, which is exactly why you must configure it carefully. Add these to settings.py:
# Tell site owners who you are and how to reach you
USER_AGENT = "BookScraperBot/1.0 (+https://example.com/bot; [email protected])"
# Adapt the delay to how quickly the server responds
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 10
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Retry temporary failures a couple of times
RETRY_TIMES = 2
# During development only: cache responses on disk so repeat runs don't hit the site
HTTPCACHE_ENABLED = True
# Export automatically, without command-line flags
FEEDS = {
"output/books.json": {"format": "json", "mode": "overwrite", "indent": 2},
}What each does:
USER_AGENTreplaces Scrapy's default. An honest, descriptive agent with contact details lets site owners reach you instead of blocking you.AutoThrottle adjusts the delay between requests based on how fast the server responds, backing off when the site slows down. It never goes below the project's
DOWNLOAD_DELAYof 1 second, so it can only make the crawl gentler than the default.HTTPCACHE_ENABLEDstores every response on disk. Re-running the spider while you tweak your selectors then costs the site nothing. Turn it off for real runs, or you'll scrape stale data.FEEDSdefines the output in settings, with"mode": "overwrite"replacing the file on each run.
Two more settings are handy while testing: CLOSESPIDER_PAGECOUNT and CLOSESPIDER_ITEMCOUNT stop the crawl after roughly that many pages or items (requests already in flight still finish). You can pass any setting on the command line with -s:
scrapy crawl books -s CLOSESPIDER_PAGECOUNT=10 -O sample.jsonOne catch: -O adds a second output on top of FEEDS, so this sample run also overwrites output/books.json with the partial result. Comment out FEEDS while you experiment if that bothers you.
The sandbox has 50 listing pages of 20 books each, so a full crawl is about 1,050 requests. With a delay of at least one second each, expect 20 minutes or more, so test with a small sample first.
Spider arguments
You can also pass arguments to a spider with -a. For example, to crawl one category instead of the whole site, accept a start URL:
class BooksSpider(scrapy.Spider):
name = "books"
allowed_domains = ["books.toscrape.com"]
start_urls = ["https://books.toscrape.com/"]
def __init__(self, start_url=None, *args, **kwargs):
super().__init__(*args, **kwargs)
if start_url:
self.start_urls = [start_url]scrapy crawl books -a start_url=https://books.toscrape.com/catalogue/category/books/travel_2/index.html -O travel.jsonReading the Results
When a crawl finishes, Scrapy prints a statistics summary. These lines tell you most of what you need:
Stat | Meaning |
|---|---|
| Items that made it through the pipelines |
| Items discarded with |
| Successful responses (look for 403, 404, 429 here too) |
| Requests skipped because the URL had been seen |
| Requests blocked by |
|
|
A stat that never happened isn't printed at all, so no robotstxt/forbidden line means nothing was blocked. Check these after every run. A crawl that "worked" but dropped half its items, or got a pile of 403 responses, is telling you something.
Running a Spider from a Script
Sometimes you want to run a crawl from your own Python code, for example from a scheduler. Put this in a file at the project root:
from scrapy.crawler import AsyncCrawlerProcess
from scrapy.utils.project import get_project_settings
process = AsyncCrawlerProcess(get_project_settings())
process.crawl("books")
process.start() # blocks until the crawl is finishedRun it from the same folder as scrapy.cfg, so Scrapy finds your project settings.
Responsible Scraping with Scrapy
Because Scrapy can send many requests quickly, a few habits matter more here than in a script:
Leave
ROBOTSTXT_OBEY = True. Turning it off to get at pages a site has asked crawlers to avoid is the wrong fix for a "Forbidden by robots.txt" message.Keep the delay and the concurrency low. The generated defaults are conservative, and AutoThrottle helps. Don't raise concurrency against a small site.
Identify yourself with
USER_AGENT, and respect429 Too Many RequestsandRetry-Afterby slowing down, not by retrying harder.Read the terms of service, and avoid personal data unless you have a lawful reason to collect it. Our guide "Is Web Scraping Legal?" covers this in depth.
Troubleshooting
Symptom | Likely cause | What to try |
|---|---|---|
Spider runs but crawls nothing ( | Code from an older tutorial defines | Use |
|
| Use |
Empty or | Selector doesn't match | Test it in |
| The link is outside | Check the domain, or leave it if it's intended |
| The site disallows that path | Respect it; don't disable |
| You're not in the project folder | Run from the folder containing |
Lots of 403 or 429 responses | You're going too fast, or the site refuses your traffic | Slow down; check the site's rules |
Fewer items than expected | Pipelines dropped some, or the duplicate filter skipped URLs | Check |
For more detail while debugging, run with -s LOG_LEVEL=DEBUG.
Where to Go from Here
You now have the core of a real crawler: a spider, items, pipelines, polite settings, and exports. Natural next steps:
CrawlSpiderrules let you describe which links to follow declaratively instead of writing the pagination code yourself.JavaScript sites need a browser. Plugins such as
scrapy-playwrightconnect Scrapy to Playwright, so you keep the framework's crawling and pipelines.Storing results in a database is a matter of writing another pipeline, for example one that inserts into SQLite or PostgreSQL.
Scheduling a nightly crawl is easy once your spider runs from a script or a command.
The mental model to remember: spiders extract, pipelines clean, settings decide how politely you crawl, and Scrapy runs the machinery. Once you've written your first spider, the second takes minutes.
Happy scraping, and please do it politely!
Found this helpful? Share it with a fellow developer or leave your questions in the comments.
Comments(0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!