{"schemaVersion":"1.0","type":"Article","types":["Article"],"slug":"python-scraping-at-scale-distributed-crawling-across-multiple-machines-2026-kti42","url":"https://zyvop.com/python-scraping-at-scale-distributed-crawling-across-multiple-machines-2026-kti42","title":"Python Scraping at Scale: Distributed Crawling Across Multiple Machines (2026)","subtitle":"Learn how to scale Python web scrapers across multiple machines in 2026 — using scrapy-redis, Docker, Kubernetes, and a production-grade proxy management layer with full working code.","tldr":"Introduction: When One Machine Is No Longer EnoughAlmost every Python developer has written a scraper. requests.get(), BeautifulSoup, a for loop, a CSV writer. ...","keywords":["python scraping kubernetes docker scale","scale python scraper cloud","scrapy cluster tutorial 2026","python distributed web scraping","distributed scraping architecture python","Scraping"],"entities":["ZyVOP","python scraping kubernetes docker scale","scale python scraper cloud","scrapy cluster tutorial 2026","python distributed web scraping","distributed scraping architecture python","Scraping"],"keyTakeaways":["Component Tool Role Spider Scrapy + scrapy-redis Crawl logic + distributed request handling Queue Redis Shared URL queue with built-in deduplication Worker deployment Docker Compose / Kubernetes Horizontal scale, stateless workers Proxy management Custom ProxyPool Health-tracked, weighted proxy selection Storage MongoDB (bulk upserts) Centralised, deduplicated results Monitoring Custom dashboard + Flower Real-time progress and worker health Resumability SCHEDULER_PERSIST=True Crash-safe, restartable crawls."],"headings":["Introduction: When One Machine Is No Longer Enough","The Architecture: One Queue, Many Workers","Part 1: Setting Up scrapy-redis","The distributed spider","settings.py for distributed mode","Part 2: Production Middlewares","Part 3: MongoDB Pipeline with Bulk Writes","Part 4: Docker Compose — Multi-Worker Setup","Part 5: The Seeder — Feeding URLs Into the Queue","Part 6: Scaling to Kubernetes","Part 7: Proxy Management at Scale","Part 8: Monitoring Your Distributed Crawl","Part 9: Resumable Crawls","Performance Numbers: What to Expect","Summary"],"outboundLinks":[],"contentText":"Introduction: When One Machine Is No Longer Enough Most Python developers write simple scrapers—requests, BeautifulSoup, a loop, CSV writer—just to get data once or twice. When scaling or running them over months, the real challenge is building a robust, scheduled system, not the parsing logic. The extraction code is tiny; the critical components are the queue, cache, storage, block detection, recovery loop, and exporters that keep the scraper alive against real‑world internet conditions. At small scale, a single async Python process can handle thousands of pages per hour. But when you need to crawl millions of pages per day — competitor catalogues, job markets, news archives, e-commerce databases — a single machine hits hard limits: one IP address, one CPU, one point of failure. Distributed scraping involves spreading tasks across multiple machines to increase speed and volume. Throttling and introducing random delays between requests can help to prevent IP bans, while rotating proxies help distribute requests and avoid detection. Managing sessions and leveraging parallel processing can further enhance efficiency. This guide shows you exactly how to build a distributed scraping system that runs across multiple machines — using scrapy-redis for shared request queues, Docker for containerisation, and a production proxy management layer that survives real-world conditions. The Architecture: One Queue, Many Workers The core insight behind distributed scraping is simple: replace Scrapy's in-memory request queue with a shared Redis queue that every worker machine can read from. ┌─────────────────────────────────────────────────────────────┐ │ DISTRIBUTED SCRAPER │ │ │ │ Master │ │ ┌──────────┐ Seeds requests ┌─────────┐ │ │ │ Spider │──────────────────────▶│ Redis │ │ │ │ (seed) │ │ Queue │ │ │ └──────────┘ └────┬────┘ │ │ │ │ │ Workers (any number of machines) │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Worker 1 │ │ Worker 2 │ │ Worker N │ ← pull jobs │ │ │ (spider) │ │ (spider) │ │ (spider) │ │ │ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ │ │ │ │ │ │ └──────────────┴──────────────┘ │ │ │ │ │ ┌─────▼──────┐ │ │ │ MongoDB │ ← all workers write here │ │ │ (results) │ │ │ └────────────┘ │ └─────────────────────────────────────────────────────────────┘ scrapy-redis lets multiple spider instances across machines pull from the same request queue with deduplication and job scheduling. This is the standard approach for large-scale web scraping that Python teams use in production. Part 1: Setting Up scrapy-redis pip install scrapy scrapy-redis redis pymongo The distributed spider # spiders/distributed_product_spider.py import scrapy from scrapy_redis.spiders import RedisSpider from urllib.parse import urljoin from datetime import datetime, timezone class DistributedProductSpider(RedisSpider): \"\"\" A Scrapy spider that pulls start URLs from a Redis list instead of a hardcoded start_urls list. To start crawling, push seed URLs to Redis: redis-cli lpush products:start_urls \"https://example-store.com/products/\" Any number of workers running this spider will cooperatively process the shared queue — each URL is processed exactly once. \"\"\" name = \"distributed_products\" redis_key = \"products:start_urls\" # Redis list to pop URLs from # How many URLs to pop from Redis at once per worker redis_batch_size = 16 # Spider-level settings — override settings.py per spider custom_settings = { \"CONCURRENT_REQUESTS\": 32, \"CONCURRENT_REQUESTS_PER_DOMAIN\": 8, \"DOWNLOAD_DELAY\": 0.5, \"RANDOMIZE_DOWNLOAD_DELAY\": True, \"AUTOTHROTTLE_ENABLED\": True, \"AUTOTHROTTLE_TARGET_CONCURRENCY\": 16.0, \"AUTOTHROTTLE_MAX_DELAY\": 5.0, \"RETRY_TIMES\": 3, \"RETRY_HTTP_CODES\": [429, 500, 502, 503, 504], } def parse(self, response): \"\"\" Parse a product listing page. Yields product detail requests AND discovers pagination links. \"\"\" # Follow product links to detail pages for href in response.css(\"a.product-link::attr(href)\").getall(): yield response.follow(href, callback=self.parse_product) # Auto-discover pagination — push next page back to Redis queue next_page = response.css(\"a[rel='next']::attr(href)\").get() if next_page: # Use follow() to handle relative URLs yield response.follow(next_page, callback=self.parse) def parse_product(self, response): \"\"\"Extract product data from a detail page.\"\"\" yield { \"url\": response.url, \"title\": response.css(\"h1::text\").get(\"\").strip(), \"price\": response.css(\".price::text\").get(\"\").strip(), \"sku\": response.css(\"[data-sku]::attr(data-sku)\").get(), \"in_stock\": bool(response.css(\".in-stock\")), \"description\":response.css(\".product-description::text\").get(\"\").strip()[:500], \"scraped_at\": datetime.now(timezone.utc).isoformat(), \"worker_id\": self.settings.get(\"WORKER_ID\", \"unknown\"), } settings.py for distributed mode # settings.py BOT_NAME = \"distributed_scraper\" SPIDER_MODULES = [\"spiders\"] # ── scrapy-redis settings ───────────────────────────────────── SCHEDULER = \"scrapy_redis.scheduler.Scheduler\" DUPEFILTER_CLASS = \"scrapy_redis.dupefilter.RFPDupeFilter\" REDIS_URL = \"redis://redis-host:6379\" # Keep crawl state across restarts — resumable crawls SCHEDULER_PERSIST = True # How long to wait for new URLs before worker shuts down SCHEDULER_IDLE_BEFORE_CLOSE = 30 # ── MongoDB pipeline ────────────────────────────────────────── ITEM_PIPELINES = { \"pipelines.MongoPipeline\": 100, \"pipelines.DuplicateFilterPipeline\": 50, } MONGO_URI = \"mongodb://mongo-host:27017/\" MONGO_DATABASE = \"distributed_scrape\" # ── Concurrency ─────────────────────────────────────────────── CONCURRENT_REQUESTS = 32 CONCURRENT_REQUESTS_PER_DOMAIN = 8 DOWNLOAD_DELAY = 0.5 RANDOMIZE_DOWNLOAD_DELAY = True # ── Retry ───────────────────────────────────────────────────── RETRY_ENABLED = True RETRY_TIMES = 3 RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524] # ── Logging ─────────────────────────────────────────────────── LOG_LEVEL = \"INFO\" # ── Downloader middlewares ──────────────────────────────────── DOWNLOADER_MIDDLEWARES = { \"scrapy.downloadermiddlewares.useragent.UserAgentMiddleware\": None, \"middlewares.RotatingProxyMiddleware\": 350, \"middlewares.UserAgentRotationMiddleware\": 400, \"scrapy.downloadermiddlewares.retry.RetryMiddleware\": 550, } # ── User agent pool ─────────────────────────────────────────── USER_AGENTS = [ \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0 Safari/537.36\", \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/119.0.0.0 Safari/537.36\", \"Mozilla/5.0 (X11; Linux x86_64) Chrome/118.0.0.0 Safari/537.36\", \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0\", ] Part 2: Production Middlewares # middlewares.py import random import logging from scrapy import signals from scrapy.exceptions import NotConfigured logger = logging.getLogger(__name__) class UserAgentRotationMiddleware: \"\"\"Rotate User-Agent on every request.\"\"\" def __init__(self, user_agents): self.user_agents = user_agents @classmethod def from_crawler(cls, crawler): agents = crawler.settings.getlist(\"USER_AGENTS\") if not agents: raise NotConfigured(\"USER_AGENTS not set\") return cls(agents) def process_request(self, request, spider): request.headers[\"User-Agent\"] = random.choice(self.user_agents) class RotatingProxyMiddleware: \"\"\" Rotate through a proxy pool. Tracks failures per proxy and removes bad proxies from the pool. \"\"\" def __init__(self, proxies, max_failures=5): self.proxies = list(proxies) self.failures = {} self.max_failures = max_failures @classmethod def from_crawler(cls, crawler): proxies = crawler.settings.getlist(\"PROXY_LIST\", []) if not proxies: raise NotConfigured(\"PROXY_LIST is empty\") return cls(proxies) def process_request(self, request, spider): if not self.proxies: return # No proxies left — run without proxy = random.choice(self.proxies) request.meta[\"proxy\"] = proxy def process_response(self, request, response, spider): if response.status in (403, 407, 429): proxy = request.meta.get(\"proxy\") self._mark_failure(proxy) return response def process_exception(self, request, exception, spider): proxy = request.meta.get(\"proxy\") self._mark_failure(proxy) def _mark_failure(self, proxy): if not proxy: return self.failures[proxy] = self.failures.get(proxy, 0) + 1 if self.failures[proxy] >= self.max_failures: if proxy in self.proxies: self.proxies.remove(proxy) logger.warning(f\"Removed bad proxy: {proxy} ({self.max_failures} failures)\") class BlockDetectionMiddleware: \"\"\" Detect common block patterns and trigger retries with a different proxy. \"\"\" BLOCK_PATTERNS = [ \"access denied\", \"captcha\", \"blocked\", \"forbidden\", \"unusual traffic\", \"robot\", \"automated queries\", \"cf-browser-verification\", \"ddos-guard\", ] def process_response(self, request, response, spider): body_lower = response.text[:2000].lower() is_blocked = ( response.status in (403, 429, 503) or any(p in body_lower for p in self.BLOCK_PATTERNS) or len(response.text) < 300 ) if is_blocked: logger.warning(f\"Block detected on {request.url} — retrying\") request.meta[\"proxy\"] = None # Force new proxy on retry request.dont_filter = True return request # Re-schedule the request return response Part 3: MongoDB Pipeline with Bulk Writes # pipelines.py import logging from datetime import datetime, timezone from pymongo import MongoClient, UpdateOne from pymongo.errors import BulkWriteError from itemadapter import ItemAdapter logger = logging.getLogger(__name__) class DuplicateFilterPipeline: \"\"\"Track seen URLs in memory to drop duplicates before DB write.\"\"\" def open_spider(self, spider): self.seen_urls = set() def process_item(self, item, spider): adapter = ItemAdapter(item) url = adapter.get(\"url\", \"\") if url in self.seen_urls: from scrapy.exceptions import DropItem raise DropItem(f\"Duplicate URL: {url}\") self.seen_urls.add(url) return item class MongoPipeline: \"\"\" Write scraped items to MongoDB using bulk operations. Upserts on URL — safe to re-run without creating duplicates. \"\"\" BULK_SIZE = 200 # Flush every 200 items COLLECTION = \"products\" def __init__(self, mongo_uri, mongo_db): self.mongo_uri = mongo_uri self.mongo_db = mongo_db self._buffer = [] @classmethod def from_crawler(cls, crawler): return cls( mongo_uri=crawler.settings.get(\"MONGO_URI\", \"mongodb://localhost:27017/\"), mongo_db =crawler.settings.get(\"MONGO_DATABASE\", \"scrapy_data\"), ) def open_spider(self, spider): self.client = MongoClient(self.mongo_uri) self.col = self.client[self.mongo_db][self.COLLECTION] self.col.create_index(\"url\", unique=True) self.items_written = 0 logger.info(f\"MongoDB connected: {self.mongo_db}.{self.COLLECTION}\") def close_spider(self, spider): if self._buffer: self._flush() self.client.close() logger.info(f\"MongoDB closed. Total items written: {self.items_written}\") def process_item(self, item, spider): self._buffer.append(dict(item)) if len(self._buffer) >= self.BULK_SIZE: self._flush() return item def _flush(self): ops = [ UpdateOne({\"url\": doc[\"url\"]}, {\"$set\": doc}, upsert=True) for doc in self._buffer ] try: result = self.col.bulk_write(ops, ordered=False) count = result.upserted_count + result.modified_count self.items_written += count logger.info(f\"Flushed {len(self._buffer)} items → MongoDB\") except BulkWriteError as e: logger.error(f\"Bulk write error: {e.details.get('writeErrors', [])[:2]}\") finally: self._buffer.clear() Part 4: Docker Compose — Multi-Worker Setup # docker-compose.yml version: \"3.9\" x-worker-base: &worker-base build: . volumes: - .:/app environment: - REDIS_URL=redis://redis:6379 - MONGO_URI=mongodb://mongo:27017/ - PROXY_LIST=${PROXY_LIST} depends_on: - redis - mongo restart: unless-stopped services: # ── Infrastructure ────────────────────────────────────────── redis: image: redis:7-alpine ports: [\"6379:6379\"] command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru volumes: [\"redis_data:/data\"] mongo: image: mongo:7 ports: [\"27017:27017\"] volumes: [\"mongo_data:/data/db\"] # ── Scrapy Workers ────────────────────────────────────────── # Run as many of these as you have cores / IPs worker-1: <<: *worker-base command: > scrapy crawl distributed_products -s WORKER_ID=worker-1 -s CONCURRENT_REQUESTS=16 environment: - REDIS_URL=redis://redis:6379 - MONGO_URI=mongodb://mongo:27017/ - WORKER_ID=worker-1 worker-2: <<: *worker-base command: > scrapy crawl distributed_products -s WORKER_ID=worker-2 -s CONCURRENT_REQUESTS=16 environment: - REDIS_URL=redis://redis:6379 - MONGO_URI=mongodb://mongo:27017/ - WORKER_ID=worker-2 worker-3: <<: *worker-base command: > scrapy crawl distributed_products -s WORKER_ID=worker-3 -s CONCURRENT_REQUESTS=16 environment: - REDIS_URL=redis://redis:6379 - MONGO_URI=mongodb://mongo:27017/ - WORKER_ID=worker-3 # ── Seed Service — pushes start URLs into Redis ───────────── seeder: <<: *worker-base command: python seeder.py restart: \"no\" # Run once then exit volumes: redis_data: mongo_data: Part 5: The Seeder — Feeding URLs Into the Queue # seeder.py import redis import time import sys from urllib.parse import urlencode REDIS_URL = \"redis://localhost:6379\" REDIS_KEY = \"products:start_urls\" def seed_from_list(urls: list[str], batch_size: int = 500): \"\"\"Push a list of start URLs into the Redis queue.\"\"\" r = redis.from_url(REDIS_URL) # Clear existing queue if resuming fresh existing = r.llen(REDIS_KEY) if existing > 0: print(f\"Queue already has {existing:,} URLs. Adding to it.\") pushed = 0 for i in range(0, len(urls), batch_size): batch = urls[i:i + batch_size] r.rpush(REDIS_KEY, *batch) pushed += len(batch) print(f\"Seeded {pushed:,}/{len(urls):,} URLs\") print(f\"\\nDone. Redis queue '{REDIS_KEY}' has {r.llen(REDIS_KEY):,} URLs.\") def seed_paginated_site( base_url: str, start_page: int = 1, end_page: int = 500, page_param: str = \"page\" ): \"\"\"Generate paginated URLs and push to Redis.\"\"\" urls = [] for page in range(start_page, end_page + 1): params = {page_param: page} urls.append(f\"{base_url}?{urlencode(params)}\") seed_from_list(urls) def monitor_queue(): \"\"\"Monitor queue depth and worker progress in real time.\"\"\" r = redis.from_url(REDIS_URL) print(\"Monitoring queue depth (Ctrl+C to stop)...\") try: while True: depth = r.llen(REDIS_KEY) seen = r.scard(f\"{REDIS_KEY}:dupefilter\") or 0 print(f\" Queue: {depth:,} pending | Seen: {seen:,} processed\", end=\"\\r\") time.sleep(2) except KeyboardInterrupt: print(\"\\nMonitor stopped.\") if __name__ == \"__main__\": mode = sys.argv[1] if len(sys.argv) > 1 else \"seed\" if mode == \"seed\": seed_paginated_site( base_url = \"https://example-store.com/products\", start_page = 1, end_page = 1000, ) elif mode == \"monitor\": monitor_queue() elif mode == \"clear\": r = redis.from_url(REDIS_URL) r.delete(REDIS_KEY) print(f\"Queue '{REDIS_KEY}' cleared.\") Part 6: Scaling to Kubernetes For truly large-scale crawling across dozens of machines, Kubernetes is the standard deployment target. The key insight: each Scrapy worker is a stateless pod that reads from the shared Redis queue. # k8s/scrapy-worker-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: scrapy-workers labels: app: scrapy-worker spec: replicas: 10 # Start with 10 workers; scale up/down with kubectl selector: matchLabels: app: scrapy-worker template: metadata: labels: app: scrapy-worker spec: containers: - name: scrapy-worker image: your-registry/scrapy-worker:latest command: - scrapy - crawl - distributed_products env: - name: REDIS_URL valueFrom: secretKeyRef: name: scraper-secrets key: redis-url - name: MONGO_URI valueFrom: secretKeyRef: name: scraper-secrets key: mongo-uri - name: WORKER_ID valueFrom: fieldRef: fieldPath: metadata.name # Pod name as worker ID resources: requests: memory: \"256Mi\" cpu: \"250m\" limits: memory: \"512Mi\" cpu: \"500m\" Scale workers up or down instantly: # Scale to 20 workers kubectl scale deployment scrapy-workers --replicas=20 # Check worker status kubectl get pods -l app=scrapy-worker # View logs from all workers kubectl logs -l app=scrapy-worker --tail=50 # Auto-scale based on Redis queue depth (requires custom metrics) kubectl autoscale deployment scrapy-workers --min=2 --max=50 Part 7: Proxy Management at Scale For serious scraping in 2026, residential proxies are almost always the safer option. Proxy quality matters far more than proxy quantity. A smaller pool of clean residential IPs usually performs much better than massive low-quality networks. Here's a production proxy manager with health checking: # proxy_manager.py import asyncio import httpx import random import time import json from dataclasses import dataclass, field from typing import Optional @dataclass class ProxyHealth: url: str success_count: int = 0 failure_count: int = 0 last_used: float = 0.0 last_success: float = 0.0 avg_response_ms: float = 0.0 is_banned: bool = False @property def success_rate(self) -> float: total = self.success_count + self.failure_count return self.success_count / total if total > 0 else 0.0 @property def score(self) -> float: \"\"\"Composite score: higher = better proxy to use.\"\"\" if self.is_banned: return 0.0 recency_bonus = max(0, 1 - (time.time() - self.last_success) / 3600) speed_score = max(0, 1 - self.avg_response_ms / 5000) return self.success_rate * 0.6 + recency_bonus * 0.2 + speed_score * 0.2 class ProxyPool: \"\"\" Intelligent proxy pool with health tracking and weighted selection. Workers register success/failure, pool learns which proxies are best. \"\"\" BAN_THRESHOLD = 0.2 # Mark as banned if success rate drops below 20% def __init__(self, proxy_urls: list[str]): self.proxies = {url: ProxyHealth(url=url) for url in proxy_urls} def get_proxy(self, strategy: str = \"weighted\") -> Optional[str]: \"\"\"Select a proxy using the specified strategy.\"\"\" available = [ p for p in self.proxies.values() if not p.is_banned ] if not available: return None if strategy == \"random\": return random.choice(available).url elif strategy == \"weighted\": # Weight by score — best proxies get used more often scores = [max(p.score, 0.01) for p in available] total = sum(scores) weights = [s / total for s in scores] return random.choices(available, weights=weights)[0].url elif strategy == \"round_robin\": # Sort by last_used timestamp — use least recently used available.sort(key=lambda p: p.last_used) return available[0].url return available[0].url def report_success(self, proxy_url: str, response_ms: float): if proxy_url in self.proxies: p = self.proxies[proxy_url] p.success_count += 1 p.last_used = time.time() p.last_success = time.time() # Rolling average response time p.avg_response_ms = (p.avg_response_ms * 0.8 + response_ms * 0.2) def report_failure(self, proxy_url: str): if proxy_url in self.proxies: p = self.proxies[proxy_url] p.failure_count += 1 p.last_used = time.time() # Auto-ban proxies with very low success rate if p.failure_count > 10 and p.success_rate < self.BAN_THRESHOLD: p.is_banned = True print(f\" Proxy banned (success rate {p.success_rate:.0%}): {proxy_url}\") def get_stats(self) -> dict: active = [p for p in self.proxies.values() if not p.is_banned] banned = [p for p in self.proxies.values() if p.is_banned] avg_sr = sum(p.success_rate for p in active) / len(active) if active else 0 return { \"total\": len(self.proxies), \"active\": len(active), \"banned\": len(banned), \"avg_success_rate\": f\"{avg_sr:.1%}\", \"best_proxy\": max(active, key=lambda p: p.score).url if active else None, } async def health_check_proxies(pool: ProxyPool, test_url: str = \"https://httpbin.org/ip\"): \"\"\" Periodically check all proxies and un-ban those that have recovered. Run this as a background task. \"\"\" async with httpx.AsyncClient(timeout=10) as client: for url, proxy in list(pool.proxies.items()): if not proxy.is_banned: continue try: start = time.time() r = await client.get(test_url, proxies={\"https\": url}) if r.status_code == 200: elapsed_ms = (time.time() - start) * 1000 proxy.is_banned = False pool.report_success(url, elapsed_ms) print(f\" Proxy recovered: {url}\") except Exception: pass # Still banned stats = pool.get_stats() print(f\"Proxy pool: {stats['active']} active, {stats['banned']} banned, \" f\"avg success rate: {stats['avg_success_rate']}\") Part 8: Monitoring Your Distributed Crawl # monitor.py — real-time crawl progress dashboard import redis import time import json from pymongo import MongoClient from datetime import datetime def live_dashboard(redis_url: str, mongo_uri: str, refresh_seconds: int = 5): \"\"\"Print a live crawl progress dashboard to the terminal.\"\"\" r = redis.from_url(redis_url) db = MongoClient(mongo_uri)[\"distributed_scrape\"] start_time = time.time() prev_count = 0 try: while True: # Queue stats queue_depth = r.llen(\"products:start_urls\") seen_count = r.scard(\"products:start_urls:dupefilter\") or 0 # DB stats items_stored = db[\"products\"].count_documents({}) items_delta = items_stored - prev_count rate_per_min = items_delta * (60 / refresh_seconds) prev_count = items_stored # Worker stats (scrapy-redis stores worker heartbeats) workers = r.smembers(\"scrapy:workers\") or set() # Elapsed elapsed = time.time() - start_time h, m = divmod(int(elapsed), 3600) m, s = divmod(m, 60) print(f\"\\033[2J\\033[H\") # Clear screen print(f\"{'═'*55}\") print(f\" DISTRIBUTED SCRAPE MONITOR — {datetime.now().strftime('%H:%M:%S')}\") print(f\"{'═'*55}\") print(f\" Runtime: {h:02d}h {m:02d}m {s:02d}s\") print(f\" Active workers:{len(workers)}\") print(f\"{'─'*55}\") print(f\" Queue depth: {queue_depth:>10,} (URLs remaining)\") print(f\" URLs seen: {seen_count:>10,} (deduplicated total)\") print(f\" Items stored: {items_stored:>10,} (in MongoDB)\") print(f\" Rate: {rate_per_min:>10.0f} items/minute\") print(f\"{'─'*55}\") if rate_per_min > 0 and queue_depth > 0: eta_mins = queue_depth / rate_per_min h2, m2 = divmod(int(eta_mins * 60), 3600) m2, s2 = divmod(m2, 60) print(f\" ETA: {h2:02d}h {m2:02d}m (estimated)\") print(f\"{'═'*55}\") time.sleep(refresh_seconds) except KeyboardInterrupt: print(\"\\nMonitor stopped.\") if __name__ == \"__main__\": live_dashboard( redis_url=\"redis://localhost:6379\", mongo_uri=\"mongodb://localhost:27017/\", ) Part 9: Resumable Crawls One of the biggest advantages of scrapy-redis is that crawls are inherently resumable. If a worker crashes or you need to add more workers mid-crawl, simply restart: # Start a fresh crawl python seeder.py seed # Launch workers (they'll pick up from where they left off if SCHEDULER_PERSIST=True) docker-compose up --scale worker=5 # Pause all workers (Ctrl+C in docker-compose) # Resume later — queue state is preserved in Redis docker-compose up --scale worker=10 # Can add more workers To completely reset a crawl: # Clear the queue and deduplication filter redis-cli del products:start_urls redis-cli del products:start_urls:dupefilter python seeder.py seed # Re-seed with fresh URLs Performance Numbers: What to Expect One machine first, then scale out. Tune Scrapy performance optimization settings before throwing hardware at the problem — raise CONCURRENT_REQUESTS, turn on AUTOTHROTTLE, and enable HTTP caching. If one server can't handle scraping millions of pages, set up a shared queue for distributed crawling across multiple workers. Real-world throughput benchmarks: Setup Pages/hour Cost estimate 1 worker, no proxy ~8,000 Free 1 worker + 10 proxies ~25,000 ~$5/day 5 workers + 50 proxies ~120,000 ~$20/day 20 workers + 200 proxies ~500,000 ~$80/day 100 workers (Kubernetes) ~2,500,000 ~$350/day Summary Component Tool Role Spider Scrapy + scrapy-redis Crawl logic + distributed request handling Queue Redis Shared URL queue with built-in deduplication Worker deployment Docker Compose / Kubernetes Horizontal scale, stateless workers Proxy management Custom ProxyPool Health-tracked, weighted proxy selection Storage MongoDB (bulk upserts) Centralised, deduplicated results Monitoring Custom dashboard + Flower Real-time progress and worker health Resumability SCHEDULER_PERSIST=True Crash-safe, restartable crawls","contentHash":"sha256:06d0ffc5409f03edd61b21d7c56f3d9978a85e8221deb1ec119cb38c188611a7","authorName":"ZyVOP","authorUrl":"https://zyvop.com/author/zyvop","authorSameAs":["https://zyvop.com","https://github.com/zyvop","https://x.com/zyvop1"],"category":"Scraping","tags":["python scraping kubernetes docker scale","scale python scraper cloud","scrapy cluster tutorial 2026","python distributed web scraping","distributed scraping architecture python"],"audience":"Senior software engineers, systems architects, and technical leads working with Scraping","tone":"Practical and evidence-based engineering guidance","readingTimeMinutes":13,"wordCount":2772,"faqs":null,"primaryTopic":"Scraping","publishedAt":"2026-07-05T05:34:00.059Z","updatedAt":"2026-09-02T00:25:00.087Z","canonicalUrl":"https://zyvop.com/python-scraping-at-scale-distributed-crawling-across-multiple-machines-2026-kti42"}