Python Scraping at Scale: Distributed Crawling Across Multiple Machines (2026)

Introduction: When One Machine Is No Longer Enough

Most Python developers write simple scrapers—requests, BeautifulSoup, a loop, CSV writer—just to get data once or twice. When scaling or running them over months, the real challenge is building a robust, scheduled system, not the parsing logic. The extraction code is tiny; the critical components are the queue, cache, storage, block detection, recovery loop, and exporters that keep the scraper alive against real‑world internet conditions.

At small scale, a single async Python process can handle thousands of pages per hour. But when you need to crawl millions of pages per day — competitor catalogues, job markets, news archives, e-commerce databases — a single machine hits hard limits: one IP address, one CPU, one point of failure.

Distributed scraping involves spreading tasks across multiple machines to increase speed and volume. Throttling and introducing random delays between requests can help to prevent IP bans, while rotating proxies help distribute requests and avoid detection. Managing sessions and leveraging parallel processing can further enhance efficiency.

This guide shows you exactly how to build a distributed scraping system that runs across multiple machines — using scrapy-redis for shared request queues, Docker for containerisation, and a production proxy management layer that survives real-world conditions.

The Architecture: One Queue, Many Workers

The core insight behind distributed scraping is simple: replace Scrapy's in-memory request queue with a shared Redis queue that every worker machine can read from.

┌─────────────────────────────────────────────────────────────┐
│                    DISTRIBUTED SCRAPER                      │
│                                                             │
│  Master                                                     │
│  ┌──────────┐    Seeds requests    ┌─────────┐             │
│  │  Spider  │──────────────────────▶│  Redis  │             │
│  │ (seed)   │                      │  Queue  │             │
│  └──────────┘                      └────┬────┘             │
│                                         │                   │
│  Workers (any number of machines)       │                   │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                  │
│  │ Worker 1 │  │ Worker 2 │  │ Worker N │  ← pull jobs     │
│  │ (spider) │  │ (spider) │  │ (spider) │                  │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘                  │
│       │              │              │                        │
│       └──────────────┴──────────────┘                       │
│                          │                                   │
│                    ┌─────▼──────┐                           │
│                    │  MongoDB   │  ← all workers write here │
│                    │ (results)  │                           │
│                    └────────────┘                           │
└─────────────────────────────────────────────────────────────┘

scrapy-redis lets multiple spider instances across machines pull from the same request queue with deduplication and job scheduling. This is the standard approach for large-scale web scraping that Python teams use in production.

Part 1: Setting Up scrapy-redis

pip install scrapy scrapy-redis redis pymongo

The distributed spider

# spiders/distributed_product_spider.py
import scrapy
from scrapy_redis.spiders import RedisSpider
from urllib.parse import urljoin
from datetime import datetime, timezone


class DistributedProductSpider(RedisSpider):
    """
    A Scrapy spider that pulls start URLs from a Redis list
    instead of a hardcoded start_urls list.

    To start crawling, push seed URLs to Redis:
        redis-cli lpush products:start_urls "https://example-store.com/products/"

    Any number of workers running this spider will cooperatively
    process the shared queue — each URL is processed exactly once.
    """

    name         = "distributed_products"
    redis_key    = "products:start_urls"   # Redis list to pop URLs from

    # How many URLs to pop from Redis at once per worker
    redis_batch_size = 16

    # Spider-level settings — override settings.py per spider
    custom_settings = {
        "CONCURRENT_REQUESTS":            32,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 8,
        "DOWNLOAD_DELAY":                 0.5,
        "RANDOMIZE_DOWNLOAD_DELAY":       True,
        "AUTOTHROTTLE_ENABLED":           True,
        "AUTOTHROTTLE_TARGET_CONCURRENCY": 16.0,
        "AUTOTHROTTLE_MAX_DELAY":         5.0,
        "RETRY_TIMES":                    3,
        "RETRY_HTTP_CODES":              [429, 500, 502, 503, 504],
    }

    def parse(self, response):
        """
        Parse a product listing page.
        Yields product detail requests AND discovers pagination links.
        """
        # Follow product links to detail pages
        for href in response.css("a.product-link::attr(href)").getall():
            yield response.follow(href, callback=self.parse_product)

        # Auto-discover pagination — push next page back to Redis queue
        next_page = response.css("a[rel='next']::attr(href)").get()
        if next_page:
            # Use follow() to handle relative URLs
            yield response.follow(next_page, callback=self.parse)

    def parse_product(self, response):
        """Extract product data from a detail page."""
        yield {
            "url":        response.url,
            "title":      response.css("h1::text").get("").strip(),
            "price":      response.css(".price::text").get("").strip(),
            "sku":        response.css("[data-sku]::attr(data-sku)").get(),
            "in_stock":   bool(response.css(".in-stock")),
            "description":response.css(".product-description::text").get("").strip()[:500],
            "scraped_at": datetime.now(timezone.utc).isoformat(),
            "worker_id":  self.settings.get("WORKER_ID", "unknown"),
        }

settings.py for distributed mode

# settings.py

BOT_NAME = "distributed_scraper"
SPIDER_MODULES = ["spiders"]

# ── scrapy-redis settings ─────────────────────────────────────
SCHEDULER            = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS     = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_URL            = "redis://redis-host:6379"

# Keep crawl state across restarts — resumable crawls
SCHEDULER_PERSIST    = True

# How long to wait for new URLs before worker shuts down
SCHEDULER_IDLE_BEFORE_CLOSE = 30

# ── MongoDB pipeline ──────────────────────────────────────────
ITEM_PIPELINES = {
    "pipelines.MongoPipeline":        100,
    "pipelines.DuplicateFilterPipeline": 50,
}
MONGO_URI      = "mongodb://mongo-host:27017/"
MONGO_DATABASE = "distributed_scrape"

# ── Concurrency ───────────────────────────────────────────────
CONCURRENT_REQUESTS              = 32
CONCURRENT_REQUESTS_PER_DOMAIN   = 8
DOWNLOAD_DELAY                   = 0.5
RANDOMIZE_DOWNLOAD_DELAY         = True

# ── Retry ─────────────────────────────────────────────────────
RETRY_ENABLED    = True
RETRY_TIMES      = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524]

# ── Logging ───────────────────────────────────────────────────
LOG_LEVEL = "INFO"

# ── Downloader middlewares ────────────────────────────────────
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
    "middlewares.RotatingProxyMiddleware": 350,
    "middlewares.UserAgentRotationMiddleware": 400,
    "scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,
}

# ── User agent pool ───────────────────────────────────────────
USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/119.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) Chrome/118.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
]

Part 2: Production Middlewares

# middlewares.py
import random
import logging
from scrapy import signals
from scrapy.exceptions import NotConfigured

logger = logging.getLogger(__name__)


class UserAgentRotationMiddleware:
    """Rotate User-Agent on every request."""

    def __init__(self, user_agents):
        self.user_agents = user_agents

    @classmethod
    def from_crawler(cls, crawler):
        agents = crawler.settings.getlist("USER_AGENTS")
        if not agents:
            raise NotConfigured("USER_AGENTS not set")
        return cls(agents)

    def process_request(self, request, spider):
        request.headers["User-Agent"] = random.choice(self.user_agents)


class RotatingProxyMiddleware:
    """
    Rotate through a proxy pool.
    Tracks failures per proxy and removes bad proxies from the pool.
    """

    def __init__(self, proxies, max_failures=5):
        self.proxies      = list(proxies)
        self.failures     = {}
        self.max_failures = max_failures

    @classmethod
    def from_crawler(cls, crawler):
        proxies = crawler.settings.getlist("PROXY_LIST", [])
        if not proxies:
            raise NotConfigured("PROXY_LIST is empty")
        return cls(proxies)

    def process_request(self, request, spider):
        if not self.proxies:
            return  # No proxies left — run without
        proxy = random.choice(self.proxies)
        request.meta["proxy"] = proxy

    def process_response(self, request, response, spider):
        if response.status in (403, 407, 429):
            proxy = request.meta.get("proxy")
            self._mark_failure(proxy)
        return response

    def process_exception(self, request, exception, spider):
        proxy = request.meta.get("proxy")
        self._mark_failure(proxy)

    def _mark_failure(self, proxy):
        if not proxy:
            return
        self.failures[proxy] = self.failures.get(proxy, 0) + 1
        if self.failures[proxy] >= self.max_failures:
            if proxy in self.proxies:
                self.proxies.remove(proxy)
                logger.warning(f"Removed bad proxy: {proxy} ({self.max_failures} failures)")


class BlockDetectionMiddleware:
    """
    Detect common block patterns and trigger retries with a different proxy.
    """

    BLOCK_PATTERNS = [
        "access denied", "captcha", "blocked", "forbidden",
        "unusual traffic", "robot", "automated queries",
        "cf-browser-verification", "ddos-guard",
    ]

    def process_response(self, request, response, spider):
        body_lower = response.text[:2000].lower()

        is_blocked = (
            response.status in (403, 429, 503) or
            any(p in body_lower for p in self.BLOCK_PATTERNS) or
            len(response.text) < 300
        )

        if is_blocked:
            logger.warning(f"Block detected on {request.url} — retrying")
            request.meta["proxy"]    = None  # Force new proxy on retry
            request.dont_filter      = True
            return request           # Re-schedule the request

        return response

Part 3: MongoDB Pipeline with Bulk Writes

# pipelines.py
import logging
from datetime import datetime, timezone
from pymongo import MongoClient, UpdateOne
from pymongo.errors import BulkWriteError
from itemadapter import ItemAdapter

logger = logging.getLogger(__name__)


class DuplicateFilterPipeline:
    """Track seen URLs in memory to drop duplicates before DB write."""

    def open_spider(self, spider):
        self.seen_urls = set()

    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        url = adapter.get("url", "")
        if url in self.seen_urls:
            from scrapy.exceptions import DropItem
            raise DropItem(f"Duplicate URL: {url}")
        self.seen_urls.add(url)
        return item


class MongoPipeline:
    """
    Write scraped items to MongoDB using bulk operations.
    Upserts on URL — safe to re-run without creating duplicates.
    """

    BULK_SIZE  = 200   # Flush every 200 items
    COLLECTION = "products"

    def __init__(self, mongo_uri, mongo_db):
        self.mongo_uri = mongo_uri
        self.mongo_db  = mongo_db
        self._buffer   = []

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get("MONGO_URI", "mongodb://localhost:27017/"),
            mongo_db =crawler.settings.get("MONGO_DATABASE", "scrapy_data"),
        )

    def open_spider(self, spider):
        self.client     = MongoClient(self.mongo_uri)
        self.col        = self.client[self.mongo_db][self.COLLECTION]
        self.col.create_index("url", unique=True)
        self.items_written = 0
        logger.info(f"MongoDB connected: {self.mongo_db}.{self.COLLECTION}")

    def close_spider(self, spider):
        if self._buffer:
            self._flush()
        self.client.close()
        logger.info(f"MongoDB closed. Total items written: {self.items_written}")

    def process_item(self, item, spider):
        self._buffer.append(dict(item))
        if len(self._buffer) >= self.BULK_SIZE:
            self._flush()
        return item

    def _flush(self):
        ops = [
            UpdateOne({"url": doc["url"]}, {"$set": doc}, upsert=True)
            for doc in self._buffer
        ]
        try:
            result = self.col.bulk_write(ops, ordered=False)
            count  = result.upserted_count + result.modified_count
            self.items_written += count
            logger.info(f"Flushed {len(self._buffer)} items → MongoDB")
        except BulkWriteError as e:
            logger.error(f"Bulk write error: {e.details.get('writeErrors', [])[:2]}")
        finally:
            self._buffer.clear()

Part 4: Docker Compose — Multi-Worker Setup

# docker-compose.yml
version: "3.9"

x-worker-base: &worker-base
  build: .
  volumes:
    - .:/app
  environment:
    - REDIS_URL=redis://redis:6379
    - MONGO_URI=mongodb://mongo:27017/
    - PROXY_LIST=${PROXY_LIST}
  depends_on:
    - redis
    - mongo
  restart: unless-stopped

services:

  # ── Infrastructure ──────────────────────────────────────────
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
    command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru
    volumes: ["redis_data:/data"]

  mongo:
    image: mongo:7
    ports: ["27017:27017"]
    volumes: ["mongo_data:/data/db"]

  # ── Scrapy Workers ──────────────────────────────────────────
  # Run as many of these as you have cores / IPs
  worker-1:
    <<: *worker-base
    command: >
      scrapy crawl distributed_products
      -s WORKER_ID=worker-1
      -s CONCURRENT_REQUESTS=16
    environment:
      - REDIS_URL=redis://redis:6379
      - MONGO_URI=mongodb://mongo:27017/
      - WORKER_ID=worker-1

  worker-2:
    <<: *worker-base
    command: >
      scrapy crawl distributed_products
      -s WORKER_ID=worker-2
      -s CONCURRENT_REQUESTS=16
    environment:
      - REDIS_URL=redis://redis:6379
      - MONGO_URI=mongodb://mongo:27017/
      - WORKER_ID=worker-2

  worker-3:
    <<: *worker-base
    command: >
      scrapy crawl distributed_products
      -s WORKER_ID=worker-3
      -s CONCURRENT_REQUESTS=16
    environment:
      - REDIS_URL=redis://redis:6379
      - MONGO_URI=mongodb://mongo:27017/
      - WORKER_ID=worker-3

  # ── Seed Service — pushes start URLs into Redis ─────────────
  seeder:
    <<: *worker-base
    command: python seeder.py
    restart: "no"   # Run once then exit

volumes:
  redis_data:
  mongo_data:

Part 5: The Seeder — Feeding URLs Into the Queue

# seeder.py
import redis
import time
import sys
from urllib.parse import urlencode

REDIS_URL   = "redis://localhost:6379"
REDIS_KEY   = "products:start_urls"

def seed_from_list(urls: list[str], batch_size: int = 500):
    """Push a list of start URLs into the Redis queue."""
    r = redis.from_url(REDIS_URL)

    # Clear existing queue if resuming fresh
    existing = r.llen(REDIS_KEY)
    if existing > 0:
        print(f"Queue already has {existing:,} URLs. Adding to it.")

    pushed = 0
    for i in range(0, len(urls), batch_size):
        batch = urls[i:i + batch_size]
        r.rpush(REDIS_KEY, *batch)
        pushed += len(batch)
        print(f"Seeded {pushed:,}/{len(urls):,} URLs")

    print(f"\nDone. Redis queue '{REDIS_KEY}' has {r.llen(REDIS_KEY):,} URLs.")


def seed_paginated_site(
    base_url: str,
    start_page: int = 1,
    end_page: int = 500,
    page_param: str = "page"
):
    """Generate paginated URLs and push to Redis."""
    urls = []
    for page in range(start_page, end_page + 1):
        params = {page_param: page}
        urls.append(f"{base_url}?{urlencode(params)}")

    seed_from_list(urls)


def monitor_queue():
    """Monitor queue depth and worker progress in real time."""
    r = redis.from_url(REDIS_URL)
    print("Monitoring queue depth (Ctrl+C to stop)...")
    try:
        while True:
            depth   = r.llen(REDIS_KEY)
            seen    = r.scard(f"{REDIS_KEY}:dupefilter") or 0
            print(f"  Queue: {depth:,} pending | Seen: {seen:,} processed", end="\r")
            time.sleep(2)
    except KeyboardInterrupt:
        print("\nMonitor stopped.")


if __name__ == "__main__":
    mode = sys.argv[1] if len(sys.argv) > 1 else "seed"

    if mode == "seed":
        seed_paginated_site(
            base_url  = "https://example-store.com/products",
            start_page = 1,
            end_page   = 1000,
        )
    elif mode == "monitor":
        monitor_queue()
    elif mode == "clear":
        r = redis.from_url(REDIS_URL)
        r.delete(REDIS_KEY)
        print(f"Queue '{REDIS_KEY}' cleared.")

Part 6: Scaling to Kubernetes

For truly large-scale crawling across dozens of machines, Kubernetes is the standard deployment target. The key insight: each Scrapy worker is a stateless pod that reads from the shared Redis queue.

# k8s/scrapy-worker-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: scrapy-workers
  labels:
    app: scrapy-worker
spec:
  replicas: 10   # Start with 10 workers; scale up/down with kubectl
  selector:
    matchLabels:
      app: scrapy-worker
  template:
    metadata:
      labels:
        app: scrapy-worker
    spec:
      containers:
        - name: scrapy-worker
          image: your-registry/scrapy-worker:latest
          command:
            - scrapy
            - crawl
            - distributed_products
          env:
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: scraper-secrets
                  key: redis-url
            - name: MONGO_URI
              valueFrom:
                secretKeyRef:
                  name: scraper-secrets
                  key: mongo-uri
            - name: WORKER_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name  # Pod name as worker ID
          resources:
            requests:
              memory: "256Mi"
              cpu:    "250m"
            limits:
              memory: "512Mi"
              cpu:    "500m"

Scale workers up or down instantly:

# Scale to 20 workers
kubectl scale deployment scrapy-workers --replicas=20

# Check worker status
kubectl get pods -l app=scrapy-worker

# View logs from all workers
kubectl logs -l app=scrapy-worker --tail=50

# Auto-scale based on Redis queue depth (requires custom metrics)
kubectl autoscale deployment scrapy-workers --min=2 --max=50

Part 7: Proxy Management at Scale

For serious scraping in 2026, residential proxies are almost always the safer option. Proxy quality matters far more than proxy quantity. A smaller pool of clean residential IPs usually performs much better than massive low-quality networks.

Here's a production proxy manager with health checking:

# proxy_manager.py
import asyncio
import httpx
import random
import time
import json
from dataclasses import dataclass, field
from typing import Optional

@dataclass
class ProxyHealth:
    url:              str
    success_count:    int   = 0
    failure_count:    int   = 0
    last_used:        float = 0.0
    last_success:     float = 0.0
    avg_response_ms:  float = 0.0
    is_banned:        bool  = False

    @property
    def success_rate(self) -> float:
        total = self.success_count + self.failure_count
        return self.success_count / total if total > 0 else 0.0

    @property
    def score(self) -> float:
        """Composite score: higher = better proxy to use."""
        if self.is_banned:
            return 0.0
        recency_bonus = max(0, 1 - (time.time() - self.last_success) / 3600)
        speed_score   = max(0, 1 - self.avg_response_ms / 5000)
        return self.success_rate * 0.6 + recency_bonus * 0.2 + speed_score * 0.2


class ProxyPool:
    """
    Intelligent proxy pool with health tracking and weighted selection.
    Workers register success/failure, pool learns which proxies are best.
    """

    BAN_THRESHOLD = 0.2   # Mark as banned if success rate drops below 20%

    def __init__(self, proxy_urls: list[str]):
        self.proxies = {url: ProxyHealth(url=url) for url in proxy_urls}

    def get_proxy(self, strategy: str = "weighted") -> Optional[str]:
        """Select a proxy using the specified strategy."""
        available = [
            p for p in self.proxies.values()
            if not p.is_banned
        ]

        if not available:
            return None

        if strategy == "random":
            return random.choice(available).url

        elif strategy == "weighted":
            # Weight by score — best proxies get used more often
            scores = [max(p.score, 0.01) for p in available]
            total  = sum(scores)
            weights = [s / total for s in scores]
            return random.choices(available, weights=weights)[0].url

        elif strategy == "round_robin":
            # Sort by last_used timestamp — use least recently used
            available.sort(key=lambda p: p.last_used)
            return available[0].url

        return available[0].url

    def report_success(self, proxy_url: str, response_ms: float):
        if proxy_url in self.proxies:
            p = self.proxies[proxy_url]
            p.success_count  += 1
            p.last_used       = time.time()
            p.last_success    = time.time()
            # Rolling average response time
            p.avg_response_ms = (p.avg_response_ms * 0.8 + response_ms * 0.2)

    def report_failure(self, proxy_url: str):
        if proxy_url in self.proxies:
            p = self.proxies[proxy_url]
            p.failure_count += 1
            p.last_used      = time.time()
            # Auto-ban proxies with very low success rate
            if p.failure_count > 10 and p.success_rate < self.BAN_THRESHOLD:
                p.is_banned = True
                print(f"  Proxy banned (success rate {p.success_rate:.0%}): {proxy_url}")

    def get_stats(self) -> dict:
        active  = [p for p in self.proxies.values() if not p.is_banned]
        banned  = [p for p in self.proxies.values() if p.is_banned]
        avg_sr  = sum(p.success_rate for p in active) / len(active) if active else 0

        return {
            "total":    len(self.proxies),
            "active":   len(active),
            "banned":   len(banned),
            "avg_success_rate": f"{avg_sr:.1%}",
            "best_proxy": max(active, key=lambda p: p.score).url if active else None,
        }


async def health_check_proxies(pool: ProxyPool, test_url: str = "https://httpbin.org/ip"):
    """
    Periodically check all proxies and un-ban those that have recovered.
    Run this as a background task.
    """
    async with httpx.AsyncClient(timeout=10) as client:
        for url, proxy in list(pool.proxies.items()):
            if not proxy.is_banned:
                continue
            try:
                start = time.time()
                r = await client.get(test_url, proxies={"https": url})
                if r.status_code == 200:
                    elapsed_ms    = (time.time() - start) * 1000
                    proxy.is_banned = False
                    pool.report_success(url, elapsed_ms)
                    print(f"  Proxy recovered: {url}")
            except Exception:
                pass   # Still banned

    stats = pool.get_stats()
    print(f"Proxy pool: {stats['active']} active, {stats['banned']} banned, "
          f"avg success rate: {stats['avg_success_rate']}")

Part 8: Monitoring Your Distributed Crawl

# monitor.py — real-time crawl progress dashboard
import redis
import time
import json
from pymongo import MongoClient
from datetime import datetime

def live_dashboard(redis_url: str, mongo_uri: str, refresh_seconds: int = 5):
    """Print a live crawl progress dashboard to the terminal."""
    r   = redis.from_url(redis_url)
    db  = MongoClient(mongo_uri)["distributed_scrape"]

    start_time   = time.time()
    prev_count   = 0

    try:
        while True:
            # Queue stats
            queue_depth  = r.llen("products:start_urls")
            seen_count   = r.scard("products:start_urls:dupefilter") or 0

            # DB stats
            items_stored = db["products"].count_documents({})
            items_delta  = items_stored - prev_count
            rate_per_min = items_delta * (60 / refresh_seconds)
            prev_count   = items_stored

            # Worker stats (scrapy-redis stores worker heartbeats)
            workers = r.smembers("scrapy:workers") or set()

            # Elapsed
            elapsed = time.time() - start_time
            h, m    = divmod(int(elapsed), 3600)
            m, s    = divmod(m, 60)

            print(f"\033[2J\033[H")   # Clear screen
            print(f"{'═'*55}")
            print(f"  DISTRIBUTED SCRAPE MONITOR — {datetime.now().strftime('%H:%M:%S')}")
            print(f"{'═'*55}")
            print(f"  Runtime:       {h:02d}h {m:02d}m {s:02d}s")
            print(f"  Active workers:{len(workers)}")
            print(f"{'─'*55}")
            print(f"  Queue depth:   {queue_depth:>10,}  (URLs remaining)")
            print(f"  URLs seen:     {seen_count:>10,}  (deduplicated total)")
            print(f"  Items stored:  {items_stored:>10,}  (in MongoDB)")
            print(f"  Rate:          {rate_per_min:>10.0f}  items/minute")
            print(f"{'─'*55}")

            if rate_per_min > 0 and queue_depth > 0:
                eta_mins = queue_depth / rate_per_min
                h2, m2   = divmod(int(eta_mins * 60), 3600)
                m2, s2   = divmod(m2, 60)
                print(f"  ETA:           {h2:02d}h {m2:02d}m  (estimated)")

            print(f"{'═'*55}")
            time.sleep(refresh_seconds)

    except KeyboardInterrupt:
        print("\nMonitor stopped.")


if __name__ == "__main__":
    live_dashboard(
        redis_url="redis://localhost:6379",
        mongo_uri="mongodb://localhost:27017/",
    )

Part 9: Resumable Crawls

One of the biggest advantages of scrapy-redis is that crawls are inherently resumable. If a worker crashes or you need to add more workers mid-crawl, simply restart:

# Start a fresh crawl
python seeder.py seed

# Launch workers (they'll pick up from where they left off if SCHEDULER_PERSIST=True)
docker-compose up --scale worker=5

# Pause all workers (Ctrl+C in docker-compose)
# Resume later — queue state is preserved in Redis
docker-compose up --scale worker=10   # Can add more workers

To completely reset a crawl:

# Clear the queue and deduplication filter
redis-cli del products:start_urls
redis-cli del products:start_urls:dupefilter
python seeder.py seed   # Re-seed with fresh URLs

Performance Numbers: What to Expect

One machine first, then scale out. Tune Scrapy performance optimization settings before throwing hardware at the problem — raise CONCURRENT_REQUESTS, turn on AUTOTHROTTLE, and enable HTTP caching. If one server can't handle scraping millions of pages, set up a shared queue for distributed crawling across multiple workers.

Real-world throughput benchmarks:

Summary