{"schemaVersion":"1.0","type":"TechArticle","types":["Article","TechArticle"],"slug":"web-scraping-with-scrapy-building-a-spider-from-scratch-nt0m5","url":"https://zyvop.com/web-scraping-with-scrapy-building-a-spider-from-scratch-nt0m5","title":"Web Scraping with Scrapy: Building a Spider from Scratch","subtitle":"Stop rewriting the same scraping plumbing. Build a Scrapy spider that crawls a whole site, cleans its data and exports it, using Scrapy 2.20.","tldr":"Your requests-and-BeautifulSoup script works, until retries, delays, duplicates and exports pile up. Scrapy handles that plumbing for you. This hands-on guide builds a bookstore crawler step by step, with pipelines, polite settings and the 2.20 changes that break older tutorials.","keywords":["Web Scraping","data","spiders","Python","scrapy","Scraping"],"entities":["Bhavya Arora","Web Scraping","data","spiders","Python","scrapy","Scraping","ZyVOP"],"keyTakeaways":["A requests and BeautifulSoup script is the right way to start scraping.","But as soon as a job grows, you find yourself writing the same plumbing again and again: a loop over pages, retries, delays, de-duplication, saving results.","Soon your 30-line script is 300 lines and mostly not about the data."],"headings":["Why Scrapy?","Setting Up","Explore the Page with the Scrapy Shell","Your First Spider","About start_urls and start()","Following Pagination","Following Links to Detail Pages","Cleaning Data with Pipelines","Crawling Politely: Settings","Spider arguments","Reading the Results","Running a Spider from a Script","Responsible Scraping with Scrapy","Troubleshooting","Where to Go from Here"],"outboundLinks":["https://books.toscrape.com/"],"contentText":"A requests and BeautifulSoup script is the right way to start scraping. But as soon as a job grows, you find yourself writing the same plumbing again and again: a loop over pages, retries, delays, de-duplication, saving results. Soon your 30-line script is 300 lines and mostly not about the data. Scrapy is a framework that supplies that plumbing. You describe what to extract in a small class called a spider, and Scrapy handles requests, concurrency, retries, throttling, robots.txt, logging, and exporting results. In this guide you'll build a complete Scrapy project from scratch. It will crawl every page of a bookstore, follow each book to its detail page, clean the data in a pipeline, and export it to JSON or CSV, politely. Version note: this guide targets Scrapy 2.20 (released November 2026), which requires Python 3.11 or newer. Scrapy has changed a lot recently, and many tutorials online are out of date. The biggest example: the old start_requests() method was removed in Scrapy 2.16, and spiders now use async def start(). If you copy a spider from an older tutorial and it crawls nothing, that's probably why. Why Scrapy? Here's what you get without writing it yourself: Need In a script In Scrapy Many requests at once Threads or asyncio by hand Built-in concurrency Retrying failures Your own loop Built-in retry middleware Being polite time.sleep() Download delay and AutoThrottle Obeying robots.txt Extra code One setting, on by default Skipping repeated URLs A set() you maintain Automatic de-duplication Cleaning and saving data Mixed into the scraping code Item pipelines and feed exports Knowing what happened print() Logs and crawl statistics When not to use it. For a single page or a quick one-off, requests and BeautifulSoup are simpler. For sites that build their content with JavaScript, you need a browser: see our guide to scraping JavaScript-heavy sites with Playwright. Scrapy shines when you need to crawl many pages reliably. Setting Up Check that you have Python 3.11 or newer, then create a virtual environment and install Scrapy: python --version # 3.11 or newer python -m venv venv source venv/bin/activate # On Windows: venv\\Scripts\\activate pip install scrapy scrapy versionNow create a project: scrapy startproject bookscraper cd bookscraperYou get this layout: bookscraper/ ├── scrapy.cfg # project marker (run commands from this folder) └── bookscraper/ ├── items.py # the shape of the data you extract ├── middlewares.py # hooks into the request/response flow ├── pipelines.py # clean, validate and store items ├── settings.py # all configuration └── spiders/ # your spiders live hereTwo things in the generated settings.py are worth noticing right away: ROBOTSTXT_OBEY = True, so Scrapy respects robots.txt by default, and a DOWNLOAD_DELAY of 1 second with one request at a time per domain. New projects start out polite. Explore the Page with the Scrapy Shell Before writing any spider, experiment on the live page. The shell downloads a URL and gives you a response object to play with: scrapy shell \"https://books.toscrape.com/\"We'll use the sandbox site books.toscrape.com, which exists for practicing scraping. Inside the shell, try: &gt;&gt;&gt; response.css(\"article.product_pod h3 a::attr(title)\").get() 'A Light in the Attic' &gt;&gt;&gt; response.css(\"p.price_color::text\").getall()[:3] ['£51.77', '£53.74', '£50.10'] &gt;&gt;&gt; response.xpath(\"//li[@class='next']/a/@href\").get() 'catalogue/page-2.html'A few things to learn from this: response.css() takes CSS selectors, extended with ::text (the text of an element) and ::attr(name) (an attribute's value). response.xpath() does the same job with XPath, which is handy for odd cases. .get() returns the first match, or None if there isn't one. .getall() returns a list of every match. The shell is the fastest way to get selectors right, so use it constantly. Type exit() to leave. Your First Spider A spider is a class with a name, a place to start, and a parse method that turns a response into data. You can generate a skeleton with scrapy genspider books books.toscrape.com, but it's short enough to write by hand. Create bookscraper/spiders/books.py: import scrapy class BooksSpider(scrapy.Spider): name = \"books\" allowed_domains = [\"books.toscrape.com\"] start_urls = [\"https://books.toscrape.com/\"] def parse(self, response): for card in response.css(\"article.product_pod\"): yield { \"title\": card.css(\"h3 a::attr(title)\").get(), \"price\": card.css(\"p.price_color::text\").get(), \"url\": response.urljoin(card.css(\"h3 a::attr(href)\").get()), }How it works: name is how you run the spider. allowed_domains stops the spider from wandering off to other sites. start_urls lists the first pages to download. Scrapy requests each one and passes the response to parse. yield hands each result back to Scrapy. A spider doesn't return a list: it yields items one at a time, and Scrapy streams them to wherever they're going. Run it and export to a file: scrapy crawl books -O books.json-O (capital) overwrites the output file, so you always start fresh. Open books.json and you'll have the first page of books. Watch out for -o. In projects created with Scrapy 2.20, the generated settings include FEED_MODE = \"create\". If the file already exists, lowercase -o then logs an error (Not writing ... because it already exists) and throws that run's items away, yet the command still exits normally, so it's easy to miss. In older setups, and outside a project, -o appends, and appending a second run to a JSON file leaves you with a corrupted file. Use -O while you're developing, or pick the behavior explicitly with the FEED_MODE setting. If you just want to try a spider quickly without creating a project, scrapy runspider books.py -O books.json runs a single file. About start_urls and start() start_urls is the simplest way to say where to begin. When you need more control, such as generating a list of URLs or adding headers, write an async def start() method instead: class BooksSpider(scrapy.Spider): name = \"books\" allowed_domains = [\"books.toscrape.com\"] async def start(self): for page in range(1, 4): url = f\"https://books.toscrape.com/catalogue/page-{page}.html\" yield scrapy.Request(url, callback=self.parse)This replaces start_requests() from older tutorials. Note that async def and yield together make it an asynchronous generator, which Scrapy knows how to run. A leftover start_requests() isn't an error: the spider simply starts, reports Crawled 0 pages, and finishes without a word of complaint. Following Pagination Scraping one page isn't much of a crawler. Add this to the end of parse to follow the \"Next\" link: next_page = response.css(\"li.next a::attr(href)\").get() if next_page: yield response.follow(next_page, callback=self.parse)response.follow() creates a new request for the link and sends its response back to parse. It handles relative URLs for you (no urljoin needed), and Scrapy automatically skips any URL it has already requested. When there's no Next link on the last page, get() returns None and the crawl ends. Following Links to Detail Pages The listing doesn't have everything: the description and product code live on each book's own page. So the spider needs two callbacks: one for listing pages and one for book pages. It will also pass the star rating (which we only read from the listing) along to the second callback. First, define the shape of your data in bookscraper/items.py. Recent Scrapy templates use a plain dataclass: from dataclasses import dataclass @dataclass class Book: title: str | None = None price: str | float | None = None rating: str | int | None = None availability: str | None = None description: str | None = None upc: str | None = None url: str | None = NoneNow rewrite the spider: import scrapy from bookscraper.items import Book class BooksSpider(scrapy.Spider): name = \"books\" allowed_domains = [\"books.toscrape.com\"] start_urls = [\"https://books.toscrape.com/\"] def parse(self, response): \"\"\"Listing page: follow every book, then the Next link.\"\"\" for card in response.css(\"article.product_pod\"): rating = card.css(\"p.star-rating::attr(class)\").get().split()[-1] # \"star-rating Three\" -&gt; \"Three\" yield response.follow( card.css(\"h3 a::attr(href)\").get(), callback=self.parse_book, cb_kwargs={\"rating\": rating}, ) next_page = response.css(\"li.next a::attr(href)\").get() if next_page: yield response.follow(next_page, callback=self.parse) def parse_book(self, response, rating): \"\"\"Book page: build the item.\"\"\" availability = \"\".join(response.css(\"p.availability::text\").getall()) yield Book( title=response.css(\"h1::text\").get(), price=response.css(\"p.price_color::text\").get(), rating=rating, availability=\" \".join(availability.split()), description=response.css(\"#product_description ~ p::text\").get(), upc=response.css(\"table tr:first-child td::text\").get(), url=response.url, )Notes on what's new: Callbacks chain the crawl. parse yields requests for book pages, and each response goes to parse_book, which yields items. A callback can yield both requests and items. cb_kwargs passes extra data to the next callback as keyword arguments. That's how rating travels from the listing page to the book page. Whitespace cleanup. The availability text has line breaks and padding around it, so we collapse it with \" \".join(text.split()). Missing data. Some pages will lack a field, a description for instance. .get() returns None instead of crashing, and the dataclass defaults to None. Cleaning Data with Pipelines Right now price is a string like £51.77 and rating is a word like Three. Item pipelines are the right place to clean and validate items, keeping that logic out of the spider. Each pipeline is a class with a process_item method. Open bookscraper/pipelines.py: import re from itemadapter import ItemAdapter from scrapy.exceptions import DropItem RATINGS = {\"One\": 1, \"Two\": 2, \"Three\": 3, \"Four\": 4, \"Five\": 5} class ValidateAndCleanPipeline: def process_item(self, item): adapter = ItemAdapter(item) if not adapter.get(\"title\") or not adapter.get(\"price\"): raise DropItem(f\"Missing title or price: {adapter.get('url')}\") adapter[\"price\"] = float(re.sub(r\"[^\\d.]\", \"\", adapter[\"price\"])) # \"£51.77\" -&gt; 51.77 adapter[\"rating\"] = RATINGS.get(adapter[\"rating\"]) # \"Three\" -&gt; 3 return item class DuplicatesPipeline: def __init__(self): self.seen = set() def process_item(self, item): upc = ItemAdapter(item)[\"upc\"] if upc in self.seen: raise DropItem(f\"Duplicate book: {upc}\") self.seen.add(upc) return itemKey ideas: ItemAdapter lets one pipeline work with dictionaries, dataclasses, or scrapy.Item objects through the same interface. Return the item to pass it along, or raise DropItem to discard it. Dropped items are logged and counted but never exported. The first pipeline drops incomplete items and converts the price and rating to numbers. The second drops repeats using the UPC, a unique product code, because the same book can appear under more than one URL. Older tutorials write process_item(self, item, spider). That still runs in 2.20 but prints a deprecation warning, so drop the spider argument in new code. Pipelines are off until you switch them on in settings.py. The numbers set the order, and lower runs first: ITEM_PIPELINES = { \"bookscraper.pipelines.ValidateAndCleanPipeline\": 300, \"bookscraper.pipelines.DuplicatesPipeline\": 400, }Crawling Politely: Settings Scrapy is fast, which is exactly why you must configure it carefully. Add these to settings.py: # Tell site owners who you are and how to reach you USER_AGENT = \"BookScraperBot/1.0 (+https://example.com/bot; contact@example.com)\" # Adapt the delay to how quickly the server responds AUTOTHROTTLE_ENABLED = True AUTOTHROTTLE_START_DELAY = 1 AUTOTHROTTLE_MAX_DELAY = 10 AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Retry temporary failures a couple of times RETRY_TIMES = 2 # During development only: cache responses on disk so repeat runs don't hit the site HTTPCACHE_ENABLED = True # Export automatically, without command-line flags FEEDS = { \"output/books.json\": {\"format\": \"json\", \"mode\": \"overwrite\", \"indent\": 2}, }What each does: USER_AGENT replaces Scrapy's default. An honest, descriptive agent with contact details lets site owners reach you instead of blocking you. AutoThrottle adjusts the delay between requests based on how fast the server responds, backing off when the site slows down. It never goes below the project's DOWNLOAD_DELAY of 1 second, so it can only make the crawl gentler than the default. HTTPCACHE_ENABLED stores every response on disk. Re-running the spider while you tweak your selectors then costs the site nothing. Turn it off for real runs, or you'll scrape stale data. FEEDS defines the output in settings, with \"mode\": \"overwrite\" replacing the file on each run. Two more settings are handy while testing: CLOSESPIDER_PAGECOUNT and CLOSESPIDER_ITEMCOUNT stop the crawl after roughly that many pages or items (requests already in flight still finish). You can pass any setting on the command line with -s: scrapy crawl books -s CLOSESPIDER_PAGECOUNT=10 -O sample.jsonOne catch: -O adds a second output on top of FEEDS, so this sample run also overwrites output/books.json with the partial result. Comment out FEEDS while you experiment if that bothers you. The sandbox has 50 listing pages of 20 books each, so a full crawl is about 1,050 requests. With a delay of at least one second each, expect 20 minutes or more, so test with a small sample first. Spider arguments You can also pass arguments to a spider with -a. For example, to crawl one category instead of the whole site, accept a start URL: class BooksSpider(scrapy.Spider): name = \"books\" allowed_domains = [\"books.toscrape.com\"] start_urls = [\"https://books.toscrape.com/\"] def __init__(self, start_url=None, *args, **kwargs): super().__init__(*args, **kwargs) if start_url: self.start_urls = [start_url]scrapy crawl books -a start_url=https://books.toscrape.com/catalogue/category/books/travel_2/index.html -O travel.jsonReading the Results When a crawl finishes, Scrapy prints a statistics summary. These lines tell you most of what you need: Stat Meaning item_scraped_count Items that made it through the pipelines item_dropped_count Items discarded with DropItem downloader/response_status_count/200 Successful responses (look for 403, 404, 429 here too) dupefilter/filtered Requests skipped because the URL had been seen robotstxt/forbidden Requests blocked by robots.txt finish_reason finished for a normal end, or the reason it stopped early A stat that never happened isn't printed at all, so no robotstxt/forbidden line means nothing was blocked. Check these after every run. A crawl that \"worked\" but dropped half its items, or got a pile of 403 responses, is telling you something. Running a Spider from a Script Sometimes you want to run a crawl from your own Python code, for example from a scheduler. Put this in a file at the project root: from scrapy.crawler import AsyncCrawlerProcess from scrapy.utils.project import get_project_settings process = AsyncCrawlerProcess(get_project_settings()) process.crawl(\"books\") process.start() # blocks until the crawl is finishedRun it from the same folder as scrapy.cfg, so Scrapy finds your project settings. Responsible Scraping with Scrapy Because Scrapy can send many requests quickly, a few habits matter more here than in a script: Leave ROBOTSTXT_OBEY = True. Turning it off to get at pages a site has asked crawlers to avoid is the wrong fix for a \"Forbidden by robots.txt\" message. Keep the delay and the concurrency low. The generated defaults are conservative, and AutoThrottle helps. Don't raise concurrency against a small site. Identify yourself with USER_AGENT, and respect 429 Too Many Requests and Retry-After by slowing down, not by retrying harder. Read the terms of service, and avoid personal data unless you have a lawful reason to collect it. Our guide \"Is Web Scraping Legal?\" covers this in depth. Troubleshooting Symptom Likely cause What to try Spider runs but crawls nothing (Crawled 0 pages) Code from an older tutorial defines start_requests(), which was removed in 2.16 Use start_urls or async def start() Not writing ... already exists and an empty export -o on an existing file; projects from 2.20 set FEED_MODE = \"create\" Use -O, or set FEED_MODE explicitly Empty or None fields Selector doesn't match Test it in scrapy shell Filtered offsite request in the log The link is outside allowed_domains Check the domain, or leave it if it's intended Forbidden by robots.txt The site disallows that path Respect it; don't disable ROBOTSTXT_OBEY ModuleNotFoundError: bookscraper You're not in the project folder Run from the folder containing scrapy.cfg Lots of 403 or 429 responses You're going too fast, or the site refuses your traffic Slow down; check the site's rules Fewer items than expected Pipelines dropped some, or the duplicate filter skipped URLs Check item_dropped_count and dupefilter/filtered For more detail while debugging, run with -s LOG_LEVEL=DEBUG. Where to Go from Here You now have the core of a real crawler: a spider, items, pipelines, polite settings, and exports. Natural next steps: CrawlSpider rules let you describe which links to follow declaratively instead of writing the pagination code yourself. JavaScript sites need a browser. Plugins such as scrapy-playwright connect Scrapy to Playwright, so you keep the framework's crawling and pipelines. Storing results in a database is a matter of writing another pipeline, for example one that inserts into SQLite or PostgreSQL. Scheduling a nightly crawl is easy once your spider runs from a script or a command. The mental model to remember: spiders extract, pipelines clean, settings decide how politely you crawl, and Scrapy runs the machinery. Once you've written your first spider, the second takes minutes. Happy scraping, and please do it politely! Found this helpful? Share it with a fellow developer or leave your questions in the comments.","contentHash":"sha256:48f0268d9061cbb3260557915dfe880601b6db847788b96d15f044fb37ad023c","authorName":"Bhavya Arora","authorUrl":"https://zyvop.com/author/bhavya","authorSameAs":[],"category":"Scraping","tags":["Web Scraping","data","spiders","Python","scrapy"],"audience":"Software engineers and developers building applications with Scraping","tone":"Practical and evidence-based engineering guidance","readingTimeMinutes":11,"wordCount":2768,"faqs":null,"primaryTopic":"Scraping","publishedAt":"2026-11-16T07:51:05.941Z","updatedAt":"2026-11-16T07:51:05.941Z","canonicalUrl":"https://zyvop.com/web-scraping-with-scrapy-building-a-spider-from-scratch-nt0m5"}