Scraping

Web Scraping with Scrapy: Building a Spider from Scratch

Stop rewriting the same scraping plumbing. Build a Scrapy spider that crawls a whole site, cleans its data and exports it, using Scrapy 2.20.

·
10 min read
·
AI-assisted

AI tooling disclosure

Generative AI materially contributed to this article.

Human review confirmed by the author.

Web Scraping with Scrapy: Building a Spider from Scratch

A requests and BeautifulSoup script is the right way to start scraping. But as soon as a job grows, you find yourself writing the same plumbing again and again: a loop over pages, retries, delays, de-duplication, saving results. Soon your 30-line script is 300 lines and mostly not about the data.

Scrapy is a framework that supplies that plumbing. You describe what to extract in a small class called a spider, and Scrapy handles requests, concurrency, retries, throttling, robots.txt, logging, and exporting results.

In this guide you'll build a complete Scrapy project from scratch. It will crawl every page of a bookstore, follow each book to its detail page, clean the data in a pipeline, and export it to JSON or CSV, politely.

Version note: this guide targets Scrapy 2.20 (released November 2026), which requires Python 3.11 or newer. Scrapy has changed a lot recently, and many tutorials online are out of date. The biggest example: the old start_requests() method was removed in Scrapy 2.16, and spiders now use async def start(). If you copy a spider from an older tutorial and it crawls nothing, that's probably why.


Why Scrapy?

Here's what you get without writing it yourself:

Need

In a script

In Scrapy

Many requests at once

Threads or asyncio by hand

Built-in concurrency

Retrying failures

Your own loop

Built-in retry middleware

Being polite

time.sleep()

Download delay and AutoThrottle

Obeying robots.txt

Extra code

One setting, on by default

Skipping repeated URLs

A set() you maintain

Automatic de-duplication

Cleaning and saving data

Mixed into the scraping code

Item pipelines and feed exports

Knowing what happened

print()

Logs and crawl statistics

When not to use it. For a single page or a quick one-off, requests and BeautifulSoup are simpler. For sites that build their content with JavaScript, you need a browser: see our guide to scraping JavaScript-heavy sites with Playwright. Scrapy shines when you need to crawl many pages reliably.

Setting Up

Check that you have Python 3.11 or newer, then create a virtual environment and install Scrapy:

python --version                 # 3.11 or newer
python -m venv venv
source venv/bin/activate         # On Windows: venv\Scripts\activate
pip install scrapy
scrapy version

Now create a project:

scrapy startproject bookscraper
cd bookscraper

You get this layout:

bookscraper/
├── scrapy.cfg                # project marker (run commands from this folder)
└── bookscraper/
    ├── items.py              # the shape of the data you extract
    ├── middlewares.py        # hooks into the request/response flow
    ├── pipelines.py          # clean, validate and store items
    ├── settings.py           # all configuration
    └── spiders/              # your spiders live here

Two things in the generated settings.py are worth noticing right away: ROBOTSTXT_OBEY = True, so Scrapy respects robots.txt by default, and a DOWNLOAD_DELAY of 1 second with one request at a time per domain. New projects start out polite.

Explore the Page with the Scrapy Shell

Before writing any spider, experiment on the live page. The shell downloads a URL and gives you a response object to play with:

scrapy shell "https://books.toscrape.com/"

We'll use the sandbox site books.toscrape.com, which exists for practicing scraping. Inside the shell, try:

>>> response.css("article.product_pod h3 a::attr(title)").get()
'A Light in the Attic'
>>> response.css("p.price_color::text").getall()[:3]
['£51.77', '£53.74', '£50.10']
>>> response.xpath("//li[@class='next']/a/@href").get()
'catalogue/page-2.html'

A few things to learn from this:

  • response.css() takes CSS selectors, extended with ::text (the text of an element) and ::attr(name) (an attribute's value).

  • response.xpath() does the same job with XPath, which is handy for odd cases.

  • .get() returns the first match, or None if there isn't one. .getall() returns a list of every match.

The shell is the fastest way to get selectors right, so use it constantly. Type exit() to leave.

Your First Spider

A spider is a class with a name, a place to start, and a parse method that turns a response into data. You can generate a skeleton with scrapy genspider books books.toscrape.com, but it's short enough to write by hand. Create bookscraper/spiders/books.py:

import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]

    def parse(self, response):
        for card in response.css("article.product_pod"):
            yield {
                "title": card.css("h3 a::attr(title)").get(),
                "price": card.css("p.price_color::text").get(),
                "url": response.urljoin(card.css("h3 a::attr(href)").get()),
            }

How it works:

  • name is how you run the spider.

  • allowed_domains stops the spider from wandering off to other sites.

  • start_urls lists the first pages to download. Scrapy requests each one and passes the response to parse.

  • yield hands each result back to Scrapy. A spider doesn't return a list: it yields items one at a time, and Scrapy streams them to wherever they're going.

Run it and export to a file:

scrapy crawl books -O books.json

-O (capital) overwrites the output file, so you always start fresh. Open books.json and you'll have the first page of books.

Watch out for -o. In projects created with Scrapy 2.20, the generated settings include FEED_MODE = "create". If the file already exists, lowercase -o then logs an error (Not writing ... because it already exists) and throws that run's items away, yet the command still exits normally, so it's easy to miss. In older setups, and outside a project, -o appends, and appending a second run to a JSON file leaves you with a corrupted file. Use -O while you're developing, or pick the behavior explicitly with the FEED_MODE setting.

If you just want to try a spider quickly without creating a project, scrapy runspider books.py -O books.json runs a single file.

About start_urls and start()

start_urls is the simplest way to say where to begin. When you need more control, such as generating a list of URLs or adding headers, write an async def start() method instead:

class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]

    async def start(self):
        for page in range(1, 4):
            url = f"https://books.toscrape.com/catalogue/page-{page}.html"
            yield scrapy.Request(url, callback=self.parse)

This replaces start_requests() from older tutorials. Note that async def and yield together make it an asynchronous generator, which Scrapy knows how to run. A leftover start_requests() isn't an error: the spider simply starts, reports Crawled 0 pages, and finishes without a word of complaint.

Following Pagination

Scraping one page isn't much of a crawler. Add this to the end of parse to follow the "Next" link:

next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

response.follow() creates a new request for the link and sends its response back to parse. It handles relative URLs for you (no urljoin needed), and Scrapy automatically skips any URL it has already requested. When there's no Next link on the last page, get() returns None and the crawl ends.

The listing doesn't have everything: the description and product code live on each book's own page. So the spider needs two callbacks: one for listing pages and one for book pages. It will also pass the star rating (which we only read from the listing) along to the second callback.

First, define the shape of your data in bookscraper/items.py. Recent Scrapy templates use a plain dataclass:

from dataclasses import dataclass


@dataclass
class Book:
    title: str | None = None
    price: str | float | None = None
    rating: str | int | None = None
    availability: str | None = None
    description: str | None = None
    upc: str | None = None
    url: str | None = None

Now rewrite the spider:

import scrapy

from bookscraper.items import Book


class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]

    def parse(self, response):
        """Listing page: follow every book, then the Next link."""
        for card in response.css("article.product_pod"):
            rating = card.css("p.star-rating::attr(class)").get().split()[-1]   # "star-rating Three" -> "Three"
            yield response.follow(
                card.css("h3 a::attr(href)").get(),
                callback=self.parse_book,
                cb_kwargs={"rating": rating},
            )

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

    def parse_book(self, response, rating):
        """Book page: build the item."""
        availability = "".join(response.css("p.availability::text").getall())
        yield Book(
            title=response.css("h1::text").get(),
            price=response.css("p.price_color::text").get(),
            rating=rating,
            availability=" ".join(availability.split()),
            description=response.css("#product_description ~ p::text").get(),
            upc=response.css("table tr:first-child td::text").get(),
            url=response.url,
        )

Notes on what's new:

  • Callbacks chain the crawl. parse yields requests for book pages, and each response goes to parse_book, which yields items. A callback can yield both requests and items.

  • cb_kwargs passes extra data to the next callback as keyword arguments. That's how rating travels from the listing page to the book page.

  • Whitespace cleanup. The availability text has line breaks and padding around it, so we collapse it with " ".join(text.split()).

  • Missing data. Some pages will lack a field, a description for instance. .get() returns None instead of crashing, and the dataclass defaults to None.

Cleaning Data with Pipelines

Right now price is a string like £51.77 and rating is a word like Three. Item pipelines are the right place to clean and validate items, keeping that logic out of the spider. Each pipeline is a class with a process_item method. Open bookscraper/pipelines.py:

import re

from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem

RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}


class ValidateAndCleanPipeline:
    def process_item(self, item):
        adapter = ItemAdapter(item)

        if not adapter.get("title") or not adapter.get("price"):
            raise DropItem(f"Missing title or price: {adapter.get('url')}")

        adapter["price"] = float(re.sub(r"[^\d.]", "", adapter["price"]))   # "£51.77" -> 51.77
        adapter["rating"] = RATINGS.get(adapter["rating"])                   # "Three" -> 3
        return item


class DuplicatesPipeline:
    def __init__(self):
        self.seen = set()

    def process_item(self, item):
        upc = ItemAdapter(item)["upc"]
        if upc in self.seen:
            raise DropItem(f"Duplicate book: {upc}")
        self.seen.add(upc)
        return item

Key ideas:

  • ItemAdapter lets one pipeline work with dictionaries, dataclasses, or scrapy.Item objects through the same interface.

  • Return the item to pass it along, or raise DropItem to discard it. Dropped items are logged and counted but never exported.

  • The first pipeline drops incomplete items and converts the price and rating to numbers. The second drops repeats using the UPC, a unique product code, because the same book can appear under more than one URL.

  • Older tutorials write process_item(self, item, spider). That still runs in 2.20 but prints a deprecation warning, so drop the spider argument in new code.

Pipelines are off until you switch them on in settings.py. The numbers set the order, and lower runs first:

ITEM_PIPELINES = {
    "bookscraper.pipelines.ValidateAndCleanPipeline": 300,
    "bookscraper.pipelines.DuplicatesPipeline": 400,
}

Crawling Politely: Settings

Scrapy is fast, which is exactly why you must configure it carefully. Add these to settings.py:

# Tell site owners who you are and how to reach you
USER_AGENT = "BookScraperBot/1.0 (+https://example.com/bot; [email protected])"

# Adapt the delay to how quickly the server responds
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 10
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

# Retry temporary failures a couple of times
RETRY_TIMES = 2

# During development only: cache responses on disk so repeat runs don't hit the site
HTTPCACHE_ENABLED = True

# Export automatically, without command-line flags
FEEDS = {
    "output/books.json": {"format": "json", "mode": "overwrite", "indent": 2},
}

What each does:

  • USER_AGENT replaces Scrapy's default. An honest, descriptive agent with contact details lets site owners reach you instead of blocking you.

  • AutoThrottle adjusts the delay between requests based on how fast the server responds, backing off when the site slows down. It never goes below the project's DOWNLOAD_DELAY of 1 second, so it can only make the crawl gentler than the default.

  • HTTPCACHE_ENABLED stores every response on disk. Re-running the spider while you tweak your selectors then costs the site nothing. Turn it off for real runs, or you'll scrape stale data.

  • FEEDS defines the output in settings, with "mode": "overwrite" replacing the file on each run.

Two more settings are handy while testing: CLOSESPIDER_PAGECOUNT and CLOSESPIDER_ITEMCOUNT stop the crawl after roughly that many pages or items (requests already in flight still finish). You can pass any setting on the command line with -s:

scrapy crawl books -s CLOSESPIDER_PAGECOUNT=10 -O sample.json

One catch: -O adds a second output on top of FEEDS, so this sample run also overwrites output/books.json with the partial result. Comment out FEEDS while you experiment if that bothers you.

The sandbox has 50 listing pages of 20 books each, so a full crawl is about 1,050 requests. With a delay of at least one second each, expect 20 minutes or more, so test with a small sample first.

Spider arguments

You can also pass arguments to a spider with -a. For example, to crawl one category instead of the whole site, accept a start URL:

class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]

    def __init__(self, start_url=None, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if start_url:
            self.start_urls = [start_url]
scrapy crawl books -a start_url=https://books.toscrape.com/catalogue/category/books/travel_2/index.html -O travel.json

Reading the Results

When a crawl finishes, Scrapy prints a statistics summary. These lines tell you most of what you need:

Stat

Meaning

item_scraped_count

Items that made it through the pipelines

item_dropped_count

Items discarded with DropItem

downloader/response_status_count/200

Successful responses (look for 403, 404, 429 here too)

dupefilter/filtered

Requests skipped because the URL had been seen

robotstxt/forbidden

Requests blocked by robots.txt

finish_reason

finished for a normal end, or the reason it stopped early

A stat that never happened isn't printed at all, so no robotstxt/forbidden line means nothing was blocked. Check these after every run. A crawl that "worked" but dropped half its items, or got a pile of 403 responses, is telling you something.

Running a Spider from a Script

Sometimes you want to run a crawl from your own Python code, for example from a scheduler. Put this in a file at the project root:

from scrapy.crawler import AsyncCrawlerProcess
from scrapy.utils.project import get_project_settings

process = AsyncCrawlerProcess(get_project_settings())
process.crawl("books")
process.start()          # blocks until the crawl is finished

Run it from the same folder as scrapy.cfg, so Scrapy finds your project settings.

Responsible Scraping with Scrapy

Because Scrapy can send many requests quickly, a few habits matter more here than in a script:

  • Leave ROBOTSTXT_OBEY = True. Turning it off to get at pages a site has asked crawlers to avoid is the wrong fix for a "Forbidden by robots.txt" message.

  • Keep the delay and the concurrency low. The generated defaults are conservative, and AutoThrottle helps. Don't raise concurrency against a small site.

  • Identify yourself with USER_AGENT, and respect 429 Too Many Requests and Retry-After by slowing down, not by retrying harder.

  • Read the terms of service, and avoid personal data unless you have a lawful reason to collect it. Our guide "Is Web Scraping Legal?" covers this in depth.

Troubleshooting

Symptom

Likely cause

What to try

Spider runs but crawls nothing (Crawled 0 pages)

Code from an older tutorial defines start_requests(), which was removed in 2.16

Use start_urls or async def start()

Not writing ... already exists and an empty export

-o on an existing file; projects from 2.20 set FEED_MODE = "create"

Use -O, or set FEED_MODE explicitly

Empty or None fields

Selector doesn't match

Test it in scrapy shell

Filtered offsite request in the log

The link is outside allowed_domains

Check the domain, or leave it if it's intended

Forbidden by robots.txt

The site disallows that path

Respect it; don't disable ROBOTSTXT_OBEY

ModuleNotFoundError: bookscraper

You're not in the project folder

Run from the folder containing scrapy.cfg

Lots of 403 or 429 responses

You're going too fast, or the site refuses your traffic

Slow down; check the site's rules

Fewer items than expected

Pipelines dropped some, or the duplicate filter skipped URLs

Check item_dropped_count and dupefilter/filtered

For more detail while debugging, run with -s LOG_LEVEL=DEBUG.

Where to Go from Here

You now have the core of a real crawler: a spider, items, pipelines, polite settings, and exports. Natural next steps:

  • CrawlSpider rules let you describe which links to follow declaratively instead of writing the pagination code yourself.

  • JavaScript sites need a browser. Plugins such as scrapy-playwright connect Scrapy to Playwright, so you keep the framework's crawling and pipelines.

  • Storing results in a database is a matter of writing another pipeline, for example one that inserts into SQLite or PostgreSQL.

  • Scheduling a nightly crawl is easy once your spider runs from a script or a command.

The mental model to remember: spiders extract, pipelines clean, settings decide how politely you crawl, and Scrapy runs the machinery. Once you've written your first spider, the second takes minutes.

Happy scraping, and please do it politely!


Found this helpful? Share it with a fellow developer or leave your questions in the comments.

Comments(0)

Join the discussion by logging into your account.

No comments yet. Be the first to comment!

Bhavya Arora

Passionate developer sharing knowledge about modern web technologies and best practices.

Subscribe to Bhavya Arora's Newsletter

Direct email dispatches when new stories are published. Zero algorithms.