--- title: "Web Scraping with Scrapy: Building a Spider from Scratch" author: "Bhavya Arora" published: "2026-11-16T07:51:05.941Z" updated: "2026-11-16T07:51:05.941Z" url: "https://zyvop.com/web-scraping-with-scrapy-building-a-spider-from-scratch-nt0m5" canonical: "https://zyvop.com/web-scraping-with-scrapy-building-a-spider-from-scratch-nt0m5" tags: ["Web Scraping","data","spiders","Python","scrapy"] --- # Web Scraping with Scrapy: Building a Spider from Scratch Stop rewriting the same scraping plumbing. Build a Scrapy spider that crawls a whole site, cleans its data and exports it, using Scrapy 2.20. ![Cover image]() A `requests` and BeautifulSoup script is the right way to start scraping. But as soon as a job grows, you find yourself writing the same plumbing again and again: a loop over pages, retries, delays, de-duplication, saving results. Soon your 30-line script is 300 lines and mostly not about the data. **Scrapy** is a framework that supplies that plumbing. You describe _what to extract_ in a small class called a spider, and Scrapy handles requests, concurrency, retries, throttling, `robots.txt`, logging, and exporting results. In this guide you'll build a complete Scrapy project from scratch. It will crawl every page of a bookstore, follow each book to its detail page, clean the data in a pipeline, and export it to JSON or CSV, politely. > **Version note:** this guide targets **Scrapy 2.20** (released November 2026), which requires **Python 3.11 or newer**. Scrapy has changed a lot recently, and many tutorials online are out of date. The biggest example: the old `start_requests()` method was **removed in Scrapy 2.16**, and spiders now use `async def start()`. If you copy a spider from an older tutorial and it crawls nothing, that's probably why. * * * ## Why Scrapy? Here's what you get without writing it yourself: | Need | In a script | In Scrapy | | --- | --- | --- | | Many requests at once | Threads or asyncio by hand | Built-in concurrency | | Retrying failures | Your own loop | Built-in retry middleware | | Being polite | `time.sleep()` | Download delay and AutoThrottle | | Obeying `robots.txt` | Extra code | One setting, on by default | | Skipping repeated URLs | A `set()` you maintain | Automatic de-duplication | | Cleaning and saving data | Mixed into the scraping code | Item pipelines and feed exports | | Knowing what happened | `print()` | Logs and crawl statistics | **When not to use it.** For a single page or a quick one-off, `requests` and BeautifulSoup are simpler. For sites that build their content with JavaScript, you need a browser: see our guide to scraping JavaScript-heavy sites with Playwright. Scrapy shines when you need to crawl _many_ pages reliably. ## Setting Up Check that you have Python 3.11 or newer, then create a virtual environment and install Scrapy: ```bash python --version # 3.11 or newer python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate pip install scrapy scrapy version ``` Now create a project: ```bash scrapy startproject bookscraper cd bookscraper ``` You get this layout: ```text bookscraper/ ├── scrapy.cfg # project marker (run commands from this folder) └── bookscraper/ ├── items.py # the shape of the data you extract ├── middlewares.py # hooks into the request/response flow ├── pipelines.py # clean, validate and store items ├── settings.py # all configuration └── spiders/ # your spiders live here ``` Two things in the generated `settings.py` are worth noticing right away: `ROBOTSTXT_OBEY = True`, so Scrapy respects `robots.txt` by default, and a `DOWNLOAD_DELAY` of 1 second with one request at a time per domain. New projects start out polite. ## Explore the Page with the Scrapy Shell Before writing any spider, experiment on the live page. The shell downloads a URL and gives you a `response` object to play with: ```bash scrapy shell "https://books.toscrape.com/" ``` We'll use the sandbox site [books.toscrape.com](), which exists for practicing scraping. Inside the shell, try: ```pycon >>> response.css("article.product_pod h3 a::attr(title)").get() 'A Light in the Attic' >>> response.css("p.price_color::text").getall()[:3] ['£51.77', '£53.74', '£50.10'] >>> response.xpath("//li[@class='next']/a/@href").get() 'catalogue/page-2.html' ``` A few things to learn from this: - `response.css()` takes CSS selectors, extended with `::text` (the text of an element) and `::attr(name)` (an attribute's value). - `response.xpath()` does the same job with XPath, which is handy for odd cases. - `.get()` returns the first match, or `None` if there isn't one. `.getall()` returns a list of every match. The shell is the fastest way to get selectors right, so use it constantly. Type `exit()` to leave. ## Your First Spider A spider is a class with a name, a place to start, and a `parse` method that turns a response into data. You can generate a skeleton with `scrapy genspider books books.toscrape.com`, but it's short enough to write by hand. Create `bookscraper/spiders/books.py`: ```python import scrapy class BooksSpider(scrapy.Spider): name = "books" allowed_domains = ["books.toscrape.com"] start_urls = ["https://books.toscrape.com/"] def parse(self, response): for card in response.css("article.product_pod"): yield { "title": card.css("h3 a::attr(title)").get(), "price": card.css("p.price_color::text").get(), "url": response.urljoin(card.css("h3 a::attr(href)").get()), } ``` How it works: - `name` is how you run the spider. - `allowed_domains` stops the spider from wandering off to other sites. - `start_urls` lists the first pages to download. Scrapy requests each one and passes the response to `parse`. - `yield` hands each result back to Scrapy. A spider doesn't `return` a list: it yields items one at a time, and Scrapy streams them to wherever they're going. Run it and export to a file: ```bash scrapy crawl books -O books.json ``` `-O` (capital) **overwrites** the output file, so you always start fresh. Open `books.json` and you'll have the first page of books. > **Watch out for** `-o`**.** In projects created with Scrapy 2.20, the generated settings include `FEED_MODE = "create"`. If the file already exists, lowercase `-o` then logs an error (`Not writing ... because it already exists`) and throws that run's items away, yet the command still exits normally, so it's easy to miss. In older setups, and outside a project, `-o` _appends_, and appending a second run to a JSON file leaves you with a corrupted file. Use `-O` while you're developing, or pick the behavior explicitly with the `FEED_MODE` setting. If you just want to try a spider quickly without creating a project, `scrapy runspider books.py -O books.json` runs a single file. ### About `start_urls` and `start()` `start_urls` is the simplest way to say where to begin. When you need more control, such as generating a list of URLs or adding headers, write an `async def start()` method instead: ```python class BooksSpider(scrapy.Spider): name = "books" allowed_domains = ["books.toscrape.com"] async def start(self): for page in range(1, 4): url = f"https://books.toscrape.com/catalogue/page-{page}.html" yield scrapy.Request(url, callback=self.parse) ``` This replaces `start_requests()` from older tutorials. Note that `async def` and `yield` together make it an asynchronous generator, which Scrapy knows how to run. A leftover `start_requests()` isn't an error: the spider simply starts, reports `Crawled 0 pages`, and finishes without a word of complaint. ## Following Pagination Scraping one page isn't much of a crawler. Add this to the end of `parse` to follow the "Next" link: ```python next_page = response.css("li.next a::attr(href)").get() if next_page: yield response.follow(next_page, callback=self.parse) ``` `response.follow()` creates a new request for the link and sends its response back to `parse`. It handles **relative URLs for you** (no `urljoin` needed), and Scrapy automatically skips any URL it has already requested. When there's no Next link on the last page, `get()` returns `None` and the crawl ends. ## Following Links to Detail Pages The listing doesn't have everything: the description and product code live on each book's own page. So the spider needs two callbacks: one for listing pages and one for book pages. It will also pass the star rating (which we only read from the listing) along to the second callback. First, define the shape of your data in `bookscraper/items.py`. Recent Scrapy templates use a plain dataclass: ```python from dataclasses import dataclass @dataclass class Book: title: str | None = None price: str | float | None = None rating: str | int | None = None availability: str | None = None description: str | None = None upc: str | None = None url: str | None = None ``` Now rewrite the spider: ```python import scrapy from bookscraper.items import Book class BooksSpider(scrapy.Spider): name = "books" allowed_domains = ["books.toscrape.com"] start_urls = ["https://books.toscrape.com/"] def parse(self, response): """Listing page: follow every book, then the Next link.""" for card in response.css("article.product_pod"): rating = card.css("p.star-rating::attr(class)").get().split()[-1] # "star-rating Three" -> "Three" yield response.follow( card.css("h3 a::attr(href)").get(), callback=self.parse_book, cb_kwargs={"rating": rating}, ) next_page = response.css("li.next a::attr(href)").get() if next_page: yield response.follow(next_page, callback=self.parse) def parse_book(self, response, rating): """Book page: build the item.""" availability = "".join(response.css("p.availability::text").getall()) yield Book( title=response.css("h1::text").get(), price=response.css("p.price_color::text").get(), rating=rating, availability=" ".join(availability.split()), description=response.css("#product_description ~ p::text").get(), upc=response.css("table tr:first-child td::text").get(), url=response.url, ) ``` Notes on what's new: - **Callbacks chain the crawl.** `parse` yields _requests_ for book pages, and each response goes to `parse_book`, which yields _items_. A callback can yield both requests and items. - `cb_kwargs` passes extra data to the next callback as keyword arguments. That's how `rating` travels from the listing page to the book page. - **Whitespace cleanup.** The availability text has line breaks and padding around it, so we collapse it with `" ".join(text.split())`. - **Missing data.** Some pages will lack a field, a description for instance. `.get()` returns `None` instead of crashing, and the dataclass defaults to `None`. ## Cleaning Data with Pipelines Right now `price` is a string like `£51.77` and `rating` is a word like `Three`. Item **pipelines** are the right place to clean and validate items, keeping that logic out of the spider. Each pipeline is a class with a `process_item` method. Open `bookscraper/pipelines.py`: ```python import re from itemadapter import ItemAdapter from scrapy.exceptions import DropItem RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5} class ValidateAndCleanPipeline: def process_item(self, item): adapter = ItemAdapter(item) if not adapter.get("title") or not adapter.get("price"): raise DropItem(f"Missing title or price: {adapter.get('url')}") adapter["price"] = float(re.sub(r"[^\d.]", "", adapter["price"])) # "£51.77" -> 51.77 adapter["rating"] = RATINGS.get(adapter["rating"]) # "Three" -> 3 return item class DuplicatesPipeline: def __init__(self): self.seen = set() def process_item(self, item): upc = ItemAdapter(item)["upc"] if upc in self.seen: raise DropItem(f"Duplicate book: {upc}") self.seen.add(upc) return item ``` Key ideas: - `ItemAdapter` lets one pipeline work with dictionaries, dataclasses, or `scrapy.Item` objects through the same interface. - **Return the item** to pass it along, or **raise** `DropItem` to discard it. Dropped items are logged and counted but never exported. - **The first pipeline** drops incomplete items and converts the price and rating to numbers. **The second** drops repeats using the UPC, a unique product code, because the same book can appear under more than one URL. - **Older tutorials** write `process_item(self, item, spider)`. That still runs in 2.20 but prints a deprecation warning, so drop the `spider` argument in new code. Pipelines are off until you switch them on in `settings.py`. The numbers set the order, and lower runs first: ```python ITEM_PIPELINES = { "bookscraper.pipelines.ValidateAndCleanPipeline": 300, "bookscraper.pipelines.DuplicatesPipeline": 400, } ``` ## Crawling Politely: Settings Scrapy is fast, which is exactly why you must configure it carefully. Add these to `settings.py`: ```python # Tell site owners who you are and how to reach you USER_AGENT = "BookScraperBot/1.0 (+https://example.com/bot; contact@example.com)" # Adapt the delay to how quickly the server responds AUTOTHROTTLE_ENABLED = True AUTOTHROTTLE_START_DELAY = 1 AUTOTHROTTLE_MAX_DELAY = 10 AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Retry temporary failures a couple of times RETRY_TIMES = 2 # During development only: cache responses on disk so repeat runs don't hit the site HTTPCACHE_ENABLED = True # Export automatically, without command-line flags FEEDS = { "output/books.json": {"format": "json", "mode": "overwrite", "indent": 2}, } ``` What each does: - `USER_AGENT` replaces Scrapy's default. An honest, descriptive agent with contact details lets site owners reach you instead of blocking you. - **AutoThrottle** adjusts the delay between requests based on how fast the server responds, backing off when the site slows down. It never goes below the project's `DOWNLOAD_DELAY` of 1 second, so it can only make the crawl gentler than the default. - `HTTPCACHE_ENABLED` stores every response on disk. Re-running the spider while you tweak your selectors then costs the site nothing. Turn it off for real runs, or you'll scrape stale data. - `FEEDS` defines the output in settings, with `"mode": "overwrite"` replacing the file on each run. Two more settings are handy while testing: `CLOSESPIDER_PAGECOUNT` and `CLOSESPIDER_ITEMCOUNT` stop the crawl after roughly that many pages or items (requests already in flight still finish). You can pass any setting on the command line with `-s`: ```bash scrapy crawl books -s CLOSESPIDER_PAGECOUNT=10 -O sample.json ``` One catch: `-O` adds a second output on top of `FEEDS`, so this sample run also overwrites `output/books.json` with the partial result. Comment out `FEEDS` while you experiment if that bothers you. The sandbox has 50 listing pages of 20 books each, so a full crawl is about 1,050 requests. With a delay of at least one second each, expect 20 minutes or more, so test with a small sample first. ### Spider arguments You can also pass arguments to a spider with `-a`. For example, to crawl one category instead of the whole site, accept a start URL: ```python class BooksSpider(scrapy.Spider): name = "books" allowed_domains = ["books.toscrape.com"] start_urls = ["https://books.toscrape.com/"] def __init__(self, start_url=None, *args, **kwargs): super().__init__(*args, **kwargs) if start_url: self.start_urls = [start_url] ``` ```bash scrapy crawl books -a start_url=https://books.toscrape.com/catalogue/category/books/travel_2/index.html -O travel.json ``` ## Reading the Results When a crawl finishes, Scrapy prints a statistics summary. These lines tell you most of what you need: | Stat | Meaning | | --- | --- | | `item_scraped_count` | Items that made it through the pipelines | | `item_dropped_count` | Items discarded with `DropItem` | | `downloader/response_status_count/200` | Successful responses (look for 403, 404, 429 here too) | | `dupefilter/filtered` | Requests skipped because the URL had been seen | | `robotstxt/forbidden` | Requests blocked by `robots.txt` | | `finish_reason` | `finished` for a normal end, or the reason it stopped early | A stat that never happened isn't printed at all, so no `robotstxt/forbidden` line means nothing was blocked. Check these after every run. A crawl that "worked" but dropped half its items, or got a pile of 403 responses, is telling you something. ## Running a Spider from a Script Sometimes you want to run a crawl from your own Python code, for example from a scheduler. Put this in a file at the project root: ```python from scrapy.crawler import AsyncCrawlerProcess from scrapy.utils.project import get_project_settings process = AsyncCrawlerProcess(get_project_settings()) process.crawl("books") process.start() # blocks until the crawl is finished ``` Run it from the same folder as `scrapy.cfg`, so Scrapy finds your project settings. ## Responsible Scraping with Scrapy Because Scrapy can send many requests quickly, a few habits matter more here than in a script: - **Leave** `ROBOTSTXT_OBEY = True`**.** Turning it off to get at pages a site has asked crawlers to avoid is the wrong fix for a "Forbidden by robots.txt" message. - **Keep the delay and the concurrency low.** The generated defaults are conservative, and AutoThrottle helps. Don't raise concurrency against a small site. - **Identify yourself** with `USER_AGENT`, and respect `429 Too Many Requests` and `Retry-After` by slowing down, not by retrying harder. - **Read the terms of service,** and avoid personal data unless you have a lawful reason to collect it. Our guide "Is Web Scraping Legal?" covers this in depth. ## Troubleshooting | Symptom | Likely cause | What to try | | --- | --- | --- | | Spider runs but crawls nothing (`Crawled 0 pages`) | Code from an older tutorial defines `start_requests()`, which was removed in 2.16 | Use `start_urls` or `async def start()` | | `Not writing ... already exists` and an empty export | `-o` on an existing file; projects from 2.20 set `FEED_MODE = "create"` | Use `-O`, or set `FEED_MODE` explicitly | | Empty or `None` fields | Selector doesn't match | Test it in `scrapy shell` | | `Filtered offsite request` in the log | The link is outside `allowed_domains` | Check the domain, or leave it if it's intended | | `Forbidden by robots.txt` | The site disallows that path | Respect it; don't disable `ROBOTSTXT_OBEY` | | `ModuleNotFoundError: bookscraper` | You're not in the project folder | Run from the folder containing `scrapy.cfg` | | Lots of 403 or 429 responses | You're going too fast, or the site refuses your traffic | Slow down; check the site's rules | | Fewer items than expected | Pipelines dropped some, or the duplicate filter skipped URLs | Check `item_dropped_count` and `dupefilter/filtered` | For more detail while debugging, run with `-s LOG_LEVEL=DEBUG`. ## Where to Go from Here You now have the core of a real crawler: a spider, items, pipelines, polite settings, and exports. Natural next steps: - `CrawlSpider` **rules** let you describe which links to follow declaratively instead of writing the pagination code yourself. - **JavaScript sites** need a browser. Plugins such as `scrapy-playwright` connect Scrapy to Playwright, so you keep the framework's crawling and pipelines. - **Storing results in a database** is a matter of writing another pipeline, for example one that inserts into SQLite or PostgreSQL. - **Scheduling** a nightly crawl is easy once your spider runs from a script or a command. The mental model to remember: **spiders extract, pipelines clean, settings decide how politely you crawl, and Scrapy runs the machinery.** Once you've written your first spider, the second takes minutes. Happy scraping, and please do it politely! * * * _Found this helpful? Share it with a fellow developer or leave your questions in the comments._