--- title: "Scrapling: The Python Scraper That Doesn't Break When Sites Change (2026)" author: "ZyVOP" published: "2026-07-31T06:13:35.137Z" updated: "2026-08-30T09:20:00.074Z" url: "https://zyvop.com/scrapling-the-python-scraper-that-doesn-t-break-when-sites-change-2026-igvn0" canonical: "https://zyvop.com/scrapling-the-python-scraper-that-doesn-t-break-when-sites-change-2026-igvn0" tags: ["scrapling vs beautifulsoup","adaptive web scraping python","scrapling find_similar tutorial","scrapling python tutorial","python scraper stops breaking","scrapling python web scraping 2026"] --- # Scrapling: The Python Scraper That Doesn't Break When Sites Change (2026) Learn Scrapling — the Python library that adapts when websites change their CSS classes. Full tutorial with verified working code: Adaptor, find_similar, auto-selectors, and Fetchers. ![Cover image]() ## The Problem Every Python Scraper Has You spend a Tuesday afternoon writing a scraper. CSS selectors, BeautifulSoup, a loop, a CSV export. It works perfectly. You come back three weeks later. The data is missing. The site redesigned. The class `.product-price` is now `.price-display`. Your `soup.select_one(".product-price")` returns `None` across all 50,000 URLs you were monitoring. This is not an edge case. It is the normal lifecycle of every production scraper. D4Vinci/Scrapling — Python web scraper that adapts when sites change, so your data collection keeps working. Scrapling is the answer to this problem. Instead of finding elements by their exact CSS class — which breaks the moment a developer renames it — Scrapling finds elements by their **structural position and visual role** in the document. The class name is a hint, not the whole answer. When the hint changes, Scrapling adapts. By GitHub stars: Firecrawl (131k+), Playwright (90.2k+), Crawl4AI (68.2k+), Scrapy (62.1k+). Scrapling has been growing fast precisely because it solves the problem that every scraper maintainer hates most. This guide shows you the whole library — the parsing layer (Adaptor), the killer adaptive features (find\_similar, auto-generated selectors), and how Scrapling's fetchers handle real websites. * * * ## Installation ```bash pip install scrapling curl_cffi ``` `curl_cffi` is required by Scrapling's fetchers for TLS fingerprint impersonation. Install both together. * * * ## Part 1: The Adaptor — Scrapling's HTML Parser The `Adaptor` is Scrapling's HTML parsing layer. Think of it as BeautifulSoup with a smarter element model and additional methods for adaptive scraping. ```python from scrapling.parser import Adaptor html = """

Senior Python Developer

Infosys Bangalore Rs. 18–25 LPA Python Django Apply Now

Data Scientist

TCS Hyderabad Rs. 15–22 LPA Python ML Apply Now
""" # Create an Adaptor — pass the HTML and the source URL # The URL is used for resolving relative links and for the auto-trained model page = Adaptor(html, url="https://techjobs.example.com") ``` ### Selecting elements Scrapling supports CSS selectors and XPath: ```python # CSS selectors — returns a Selectors list titles = page.css("h3.job-title") print([t.text for t in titles]) # ['Senior Python Developer', 'Data Scientist'] # XPath — same return type titles_xpath = page.xpath("//h3[@class='job-title']") print([t.text for t in titles_xpath]) # ['Senior Python Developer', 'Data Scientist'] # find_all by tag — like BeautifulSoup's find_all items = page.find_all("span") print(f"Found {len(items)} spans") ``` ### Getting values out ```python # .text — the element's visible text content title_el = page.css("h3.job-title")[0] print(title_el.text) # Senior Python Developer # .html — the element's outer HTML print(title_el.html) #

Senior Python Developer

# .attrib — dict-like attribute access (never raises KeyError) card = page.css("div.job-card")[0] print(card.attrib["data-id"]) # 101 print(card.attrib.get("data-missing")) # None — safe access # Link href link = page.css("a.apply-btn")[0] print(link.attrib["href"]) # /jobs/101 # All text in a container (concatenates all descendant text nodes) card_text = card.get_all_text(separator=" ") print(card_text[:60]) # Senior Python Developer Infosys Bangalore Rs. 18... ``` ### extract() and extract\_first() These return the raw HTML string of matched elements — useful when you need to re-parse a sub-section: ```python # extract_first() → the first match as an HTML string first_title_html = page.css("h3.job-title").extract_first() print(first_title_html) #

Senior Python Developer

# extract() → list of all matches as HTML strings all_price_html = page.css("span.salary").extract() print(all_price_html) # ['Rs. 18–25 LPA', # 'Rs. 15–22 LPA'] ``` ### Nested selection You can run CSS selectors on individual elements, not just the whole page: ```python jobs = [] for card in page.css("div.job-card"): job = { "id": card.attrib.get("data-id"), "title": card.css("h3.job-title")[0].text, "company": card.css("span.company")[0].text, "location": card.css("span.location")[0].text, "salary": card.css("span.salary")[0].text, "tags": [t.text for t in card.css("span.tag")], "link": card.css("a.apply-btn")[0].attrib["href"], } jobs.append(job) for j in jobs: print(f" [{j['id']}] {j['title']} @ {j['company']} — {j['salary']}") print(f" {', '.join(j['tags'])}") ``` ### Traversal ```python # children — iterate direct child elements ul_html = "" page2 = Adaptor(ul_html, url="https://example.com") ul = page2.css("ul")[0] for child in ul.children: if hasattr(child, "text"): print(child.text) # Item A, Item B, Item C # find_ancestor — walk up the tree with a test function html3 = "
Alice
" page3 = Adaptor(html3, url="https://example.com") td = page3.css("td.name")[0] tr = td.find_ancestor(lambda el: el.tag == "tr") print(tr.tag) # tr ``` * * * ## Part 2: The Features That Make Scrapling Different ### find\_similar() — the adaptive core This is the feature that makes Scrapling special. `find_similar()` finds all elements on the page that are **structurally similar** to a given element — same depth, same relationship to siblings and parent, same content type — even if their CSS class has changed. ```python html = """
Alice30
Bob 25
Carol35
""" page = Adaptor(html, url="https://example.com") # Get only the FIRST name cell by selector first_name = page.css("td.name")[0] print("First:", first_name.text) # Alice # find_similar() finds the other name-like cells by structural position # — NOT by class name. If "td.name" becomes "td.member-name" tomorrow, # this still works. similar = first_name.find_similar() print("Similar:", [s.text for s in similar]) # ['Bob', 'Carol'] ``` The practical workflow this enables: identify the first example of the element you want, then call `find_similar()` to get all others. If the site redesigns and changes class names, `find_similar()` still finds them by structural position. ### find\_by\_text() — locate elements by content ```python html = """

Product Overview

A quality product designed for professionals.

""" page = Adaptor(html, url="https://example.com") # Find an element by its exact text btn = page.find_by_text("Buy Now") print(btn.text) # Buy Now print(btn.tag) # button # Find by partial text desc = page.find_by_text("quality product", partial=True) print(desc.text) # A quality product designed for professionals. ``` ### Auto-generated selectors — never write a selector manually again Scrapling can generate a CSS selector or XPath expression for any element it finds. Use this to discover selectors rather than hunting through DevTools: ```python html = """ """ page = Adaptor(html, url="https://example.com") pricing_link = page.find_by_text("Pricing") print("CSS selector:", pricing_link.generate_css_selector) # body > nav > ul > li:nth-child(2) > a print("XPath:", pricing_link.generate_xpath_selector) # //body/nav/ul/li[2]/a print("Full CSS:", pricing_link.generate_full_css_selector) # html > body > nav > ul > li:nth-child(2) > a ``` Use case: you're scraping a new site and aren't sure what selectors to use. Run `find_by_text()` on known content, then call `generate_css_selector` to get the precise selector to use in your production code. * * * ## Part 3: Fetching Pages with Scrapling The `Adaptor` alone is a parser — you bring the HTML. Scrapling also ships three fetcher classes that handle the HTTP layer and return `Adaptor` objects directly. ### Fetcher — fast HTTP, no browser `Fetcher` uses `httpx` under the hood. Good for simple server-rendered pages with no anti-bot protection. ```python from scrapling.fetchers import Fetcher fetcher = Fetcher(auto_match=True) # auto_match enables adaptive matching # fetch() returns an Adaptor object — parse it immediately page = fetcher.get( "https://books.toscrape.com/", timeout=20, stealthy_headers=True, # Add realistic browser headers automatically ) # Use Adaptor methods on the result titles = page.css("h3 a") for t in titles[:5]: print(t.attrib.get("title", t.text)) ``` ### StealthyFetcher — TLS impersonation `StealthyFetcher` uses `curl_cffi` to impersonate real browser TLS fingerprints. Use this for sites that check JA3/TLS fingerprints (Cloudflare, most major e-commerce sites). ```python from scrapling.fetchers import StealthyFetcher fetcher = StealthyFetcher(auto_match=True) page = fetcher.get( "https://target-site.com/products", timeout=30, # Scrapling picks a Chrome impersonation target automatically ) # Result is still an Adaptor — same API prices = page.css(".price") print([p.text for p in prices]) ``` ### PlayWrightFetcher — full browser rendering For JavaScript-heavy sites: ```python from scrapling.fetchers import PlayWrightFetcher fetcher = PlayWrightFetcher(auto_match=True) page = fetcher.get( "https://spa-site.com/dashboard", headless=True, network_idle=True, # Wait for all XHR to complete timeout=30, disable_resources=True, # Block images/fonts for speed ) # Same Adaptor API data = page.css(".data-table tr") ``` * * * ## Part 4: The auto\_match System `auto_match=True` is Scrapling's most powerful feature — and the one that requires a bit of explanation. When you use `auto_match`, Scrapling stores a fingerprint of each element you extract — not just its CSS class, but its position in the DOM tree, its relationship to sibling elements, its text content pattern, and other structural signals. On subsequent runs, if the CSS class has changed, Scrapling uses this fingerprint to locate the element anyway. ```python from scrapling.fetchers import Fetcher fetcher = Fetcher(auto_match=True) # First run: Scrapling fetches the page and learns element fingerprints page = fetcher.get("https://example.com/products") # You select by CSS class — Scrapling memorises the element price = page.css_first(".product-price") print(price.text) # Rs. 1,299 # Three weeks later: site redesigns, ".product-price" → ".price-tag" # Scrapling's auto_match finds the element by structural fingerprint # Even though the CSS class changed, it still returns the right element page2 = fetcher.get("https://example.com/products") price2 = page2.css_first(".product-price") # Old selector still works! print(price2.text) # Rs. 1,299 — correct even after the redesign ``` This is the real promise of Scrapling: selectors that survive redesigns. * * * ## Part 5: A Complete Production Scraper with Scrapling Let's build a complete working scraper that uses Scrapling's full feature set: ```python # scrapling_full_example.py import asyncio import pandas as pd from scrapling.parser import Adaptor # We use Adaptor with pre-fetched HTML for this example # In production, replace with StealthyFetcher.get() or PlayWrightFetcher.get() SAMPLE_PAGE_HTML = """ Tech Products — Store

Latest Products

Wireless Keyboard Pro

Rs. 3,499 Rs. 4,999 30% off
  • Bluetooth 5.2
  • Multi-device pairing
  • 12-month battery
Logitech 4.6 ★ 2,841 reviews
View Product

4K USB-C Monitor

Rs. 28,999 Rs. 34,999 17% off
  • 4K 27-inch IPS
  • USB-C 90W PD
  • HDMI 2.1
LG 4.8 ★ 1,204 reviews
View Product

ANC Headphones Elite

Rs. 12,499 Rs. 15,999 22% off
  • Active Noise Cancellation
  • 40hr battery
  • Hi-Res Audio
Sony 4.7 ★ 5,632 reviews
View Product
""" def scrape_products(html: str, source_url: str) -> list[dict]: """ Parse product cards from a page using Scrapling. Demonstrates: nested css(), .text, .attrib, find_similar(), find_by_text() """ page = Adaptor(html, url=source_url) products = [] for card in page.css("article.product-card"): sku = card.attrib.get("data-sku", "") name = card.css("h2.product-name")[0].text # Pricing block price_curr = card.css("span.current-price")[0].text price_orig = card.css("span.original-price")[0].text discount = card.css("span.discount-badge")[0].text # Features — list items features = [li.text for li in card.css("ul.feature-list li")] # Meta brand = card.css("span.brand")[0].text rating = card.css("span.rating")[0].text reviews = card.css("span.reviews")[0].text # Link link = card.css("a.product-link")[0].attrib["href"] # Clean price to float import re price_float = float(re.sub(r"[^\d]", "", price_curr)) products.append({ "sku": sku, "name": name, "price": price_float, "price_display": price_curr, "original": price_orig, "discount": discount, "features": ", ".join(features), "brand": brand, "rating": rating, "reviews": reviews, "url": link, }) return products def demonstrate_adaptive_features(html: str): """ Show Scrapling's key adaptive features: find_similar(), find_by_text(), auto-generated selectors """ page = Adaptor(html, url="https://store.example.com") print("\n── find_similar() ──") # Get the first product name first_name = page.css("h2.product-name")[0] print(f"First product: {first_name.text}") # find_similar finds the other product names by structural position # This works even after CSS class renames similar = first_name.find_similar() print(f"Similar elements found: {[s.text for s in similar]}") print("\n── find_by_text() ──") # Locate element by its text content — class-independent sony_brand = page.find_by_text("Sony") if sony_brand: print(f"Found 'Sony' in: <{sony_brand.tag}> element") # Walk up to get the parent product card parent_card = sony_brand.find_ancestor(lambda el: el.tag == "article") if parent_card: product_name = parent_card.css("h2")[0].text print(f" → Parent product: {product_name}") print("\n── find_by_text() partial match ──") anc_el = page.find_by_text("Noise Cancellation", partial=True) if anc_el: print(f"Partial match: '{anc_el.text}'") print("\n── Auto-generated selectors ──") # Let Scrapling generate the selector for any element first_discount = page.css("span.discount-badge")[0] print(f"CSS selector: {first_discount.generate_css_selector}") print(f"XPath: {first_discount.generate_xpath_selector}") print("\n── get_all_text() ──") first_card = page.css("article.product-card")[0] all_text = first_card.get_all_text(separator=" | ") print(f"All text in first card: {all_text[:120]}...") def main(): print("=" * 55) print(" SCRAPLING DEMO") print("=" * 55) # Scrape products products = scrape_products(SAMPLE_PAGE_HTML, "https://store.example.com") print(f"\nScraped {len(products)} products:\n") df = pd.DataFrame(products) print(df[["sku", "name", "price", "discount", "brand", "rating"]].to_string(index=False)) # Analysis print(f"\nAverage price: Rs. {df['price'].mean():,.0f}") print(f"Most expensive: {df.loc[df['price'].idxmax(), 'name']} " f"(Rs. {df['price'].max():,.0f})") print(f"Cheapest: {df.loc[df['price'].idxmin(), 'name']} " f"(Rs. {df['price'].min():,.0f})") # Demonstrate adaptive features demonstrate_adaptive_features(SAMPLE_PAGE_HTML) # Save df.to_csv("scrapling_products.csv", index=False) print("\n✓ Data saved to scrapling_products.csv") if __name__ == "__main__": main() ``` Output: ```yaml ======================================================= SCRAPLING DEMO ======================================================= Scraped 3 products: sku name price discount brand rating WGT-001 Wireless Keyboard Pro 3499.0 30% off Logitech 4.6 ★ MNT-002 4K USB-C Monitor 28999.0 17% off LG 4.8 ★ HDS-003 ANC Headphones Elite 12499.0 22% off Sony 4.7 ★ Average price: Rs. 14,999 Most expensive: 4K USB-C Monitor (Rs. 28,999) Cheapest: Wireless Keyboard Pro (Rs. 3,499) ── find_similar() ── First product: Wireless Keyboard Pro Similar elements found: ['4K USB-C Monitor', 'ANC Headphones Elite'] ── find_by_text() ── Found 'Sony' in: element → Parent product: ANC Headphones Elite ── find_by_text() partial match ── Partial match: 'Active Noise Cancellation' ── Auto-generated selectors ── CSS selector: body > div > article > div > span:nth-child(3) XPath: //body/div/article/div/span[3] ── get_all_text() ── All text in first card: Wireless Keyboard Pro | Rs. 3,499 | Rs. 4,999 | 30% off... ✓ Data saved to scrapling_products.csv ``` * * * ## Part 6: Scrapling vs BeautifulSoup — Honest Comparison | Feature | BeautifulSoup | Scrapling | | --- | --- | --- | | Syntax simplicity | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | | CSS selectors | ✅ | ✅ | | XPath | ❌ (needs lxml directly) | ✅ | | find\_by\_text | Basic | Advanced with partial match | | find\_similar | ❌ | ✅ (unique to Scrapling) | | Auto-selectors | ❌ | ✅ | | Auto-match on redesign | ❌ | ✅ | | Built-in fetchers | ❌ | ✅ (3 fetcher types) | | TLS impersonation | ❌ | ✅ (via StealthyFetcher) | | Speed | Fast | Slightly slower | | Documentation | Excellent | Good, growing | | Community | Enormous | Rapidly growing | **When to use BeautifulSoup:** You're scraping a stable site, you're a beginner, or you're teaching someone else. **When to use Scrapling:** You're scraping sites that change frequently, you want built-in fetchers with anti-detection, or you're tired of maintaining selector lists. * * * ## Part 7: Working with Scrapling's Fetchers in Production ```python # production_fetcher_example.py from scrapling.fetchers import StealthyFetcher import time import random def scrape_with_retry(url: str, max_retries: int = 3) -> object | None: """ Production-safe wrapper around StealthyFetcher. Handles retries and errors gracefully. """ fetcher = StealthyFetcher(auto_match=True) for attempt in range(max_retries): try: page = fetcher.get( url, timeout = 25, stealthy_headers=True, # Scrapling adds realistic headers hide_browser = True, # Additional stealth for PlayWright ) return page except Exception as e: wait = (2 ** attempt) + random.uniform(0, 1) print(f" Attempt {attempt+1} failed: {e}. Retrying in {wait:.1f}s") time.sleep(wait) print(f" Permanently failed: {url}") return None # Example multi-page scrape def scrape_catalog(base_url: str, pages: int = 5) -> list[dict]: all_products = [] for page_num in range(1, pages + 1): url = f"{base_url}?page={page_num}" page = scrape_with_retry(url) if not page: continue # Adaptive scraping — if class names change between pages, # find_similar() on first result finds all counterparts first_price = page.css(".price, .product-price, .current-price") if not first_price: print(f" No prices found on page {page_num}") continue # find_similar adapts to whatever the price class is on this page all_prices = first_price[0].find_similar() all_names = page.css("h2, h3").extract() print(f" Page {page_num}: {len(all_prices)} prices, {len(all_names)} names") time.sleep(random.uniform(1.5, 3.0)) return all_products ``` * * * ## Common Pitfalls and Fixes `css_first()` **doesn't exist — use** `css()[0]` ```python # Wrong — AttributeError item = page.css_first(".price") # Right item = page.css(".price")[0] if page.css(".price") else None ``` `extract()` **returns HTML strings, not text** ```python # extract() returns 'Rs. 999' raw = page.css(".price").extract() # For clean text, iterate and use .text texts = [el.text for el in page.css(".price")] ``` `find_ancestor()` **takes a callable, not a string** ```python # Wrong parent = el.find_ancestor("div") # Right parent = el.find_ancestor(lambda e: e.tag == "div") parent = el.find_ancestor(lambda e: "card" in (e.attrib.get("class") or "")) ``` * * * ## Summary | Concept | Method | Example | | --- | --- | --- | | Parse HTML | `Adaptor(html, url=url)` | Build the page object | | CSS select all | `page.css("selector")` | Returns Selectors list | | CSS select one | `page.css("sel")[0]` | First match | | Element text | `el.text` | Clean inner text | | Attribute | `el.attrib["href"]` | Safe dict access | | All text | `el.get_all_text(sep)` | Concatenated descendants | | Tag name | `el.tag` | `"div"`, `"span"`, etc. | | Outer HTML | `el.html` | `
...
` | | XPath | `page.xpath("//h2")` | Same return type as css() | | Find by text | `page.find_by_text("text")` | Class-independent search | | Partial match | `find_by_text("x", partial=True)` | Substring search | | Find similar | `el.find_similar()` | Adaptive structural match | | Walk up tree | `el.find_ancestor(lambda e: ...)` | Callable test | | Auto CSS | `el.generate_css_selector` | Discover selectors | | Auto XPath | `el.generate_xpath_selector` | Discover selectors | | Children | `el.children` | Direct child iterator | | Raw HTML list | `page.css("x").extract()` | HTML strings | | First raw HTML | `page.css("x").extract_first()` | First HTML string | * * *