--- title: "Scraping JavaScript-Heavy Websites with Playwright in Python" author: "Bhavya Arora" published: "2026-11-15T09:20:00.043Z" updated: "2026-11-14T09:17:33.377Z" url: "https://zyvop.com/scraping-javascript-heavy-websites-with-playwright-in-python-5bo7x" canonical: "https://zyvop.com/scraping-javascript-heavy-websites-with-playwright-in-python-5bo7x" tags: ["Web Scraping","BeautifulSoup","Playwright","Python","JavaScript"] --- # Scraping JavaScript-Heavy Websites with Playwright in Python Learn to scrape JavaScript-heavy sites in Python with Playwright: smart waiting, pagination, infinite scroll, JSON capture, and BeautifulSoup parsing. ![Cover image]() You write a clean `requests` and BeautifulSoup scraper, point it at a page that clearly shows a list of items in your browser, and get back... nothing. An empty list, or a page full of placeholders. That's the signature of a **JavaScript-rendered site**. The server sends a nearly empty HTML shell, and JavaScript running in your browser fills in the content afterwards. `requests` never runs that JavaScript, so it never sees the data. In this guide you'll learn how to diagnose the problem, decide whether you need a browser at all, and use **Playwright** to scrape these sites, including pagination, infinite scroll, and capturing the JSON the page loads. We'll finish with a complete script that saves results to CSV. > **Before you scrape:** check the site's terms of service and `robots.txt`, and read our guide "Is Web Scraping Legal?" first if you haven't. A real browser makes it easy to do things a site's owner may not welcome, so the responsible-use section near the end matters even more here. * * * ## Why requests and BeautifulSoup Come Back Empty Two views of the same page are easy to confuse: - **View Source** shows exactly what the server sent. That's what `requests` gets. - **Inspect Element** shows the live page _after_ JavaScript has run. If the data appears in Inspect but not in View Source, JavaScript is building it. You can confirm with a quick test against the practice site [quotes.toscrape.com](), which renders its quotes with JavaScript: ```python import requests from bs4 import BeautifulSoup html = requests.get("http://quotes.toscrape.com/js/", timeout=10).text soup = BeautifulSoup(html, "html.parser") print(len(soup.select(".quote"))) # 0: the quotes aren't in the raw HTML ``` Zero results from a selector you can clearly see in your browser's Inspect panel is the telltale sign. ## Do You Need a Browser at All? A real browser is the heaviest tool in the box. Before you reach for it, check where the data actually comes from: | Where the data lives | Best tool | | --- | --- | | In the raw HTML (View Source) | `requests` + BeautifulSoup | | Loaded from a JSON endpoint you can see in the browser's **Network** tab | Call that endpoint directly with `requests` | | Only exists after JavaScript runs, or needs scrolling, clicking, or other interaction | **Playwright** | The second row is worth a look every time, because it's often the cleanest solution. Open your browser's developer tools, switch to the **Network** tab, reload the page, and filter by **Fetch/XHR**. On our practice site the infinite-scroll page loads its quotes from a JSON endpoint: ```python import requests response = requests.get( "http://quotes.toscrape.com/api/quotes", params={"page": 1}, timeout=10 ) data = response.json() print(len(data["quotes"]), data["has_next"]) # 10 True ``` No browser needed. But sometimes the endpoint needs headers or tokens that the page generates, or the content only appears after a series of clicks. That's where Playwright earns its keep. ## Setting Up Playwright [Playwright]() is a browser automation library from Microsoft. It drives real Chromium, Firefox, or WebKit browsers, so JavaScript runs exactly as it would for a human visitor. ```bash python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate pip install playwright beautifulsoup4 requests playwright install chromium ``` Two notes: - `playwright install chromium` downloads a browser build that Playwright manages itself. Without that step, the first launch fails. - On a bare Linux server or in CI, use `playwright install --with-deps chromium` instead, which also installs the operating-system libraries the browser needs. ## Your First Playwright Scrape ```python from bs4 import BeautifulSoup from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch() # headless by default: no window opens page = browser.new_page() page.goto("http://quotes.toscrape.com/js/") page.wait_for_selector(".quote") # wait until JavaScript has rendered the quotes html = page.content() # the page *after* JavaScript ran browser.close() soup = BeautifulSoup(html, "html.parser") print(len(soup.select(".quote"))) # 10 ``` What happened: 1. Playwright launched a real browser in the background (**headless** means no visible window). 2. `page.goto()` loaded the page, and the browser ran its JavaScript. 3. `page.wait_for_selector(".quote")` paused until at least one quote existed in the page. 4. `page.content()` returned the rendered HTML, which we handed to BeautifulSoup. The parsing code is the same as in any other scraper. ## Waiting the Right Way Timing is the heart of scraping dynamic sites. Grab the HTML too early and you get an empty page, which is the most common beginner bug. **Avoid fixed sleeps.** `time.sleep(5)` is either too short (empty results) or too long (slow scraping). Wait for the thing you actually need: ```python # Fragile: guesses how long rendering takes page.goto(url) time.sleep(5) # Better: wait for the element you need page.goto(url) page.wait_for_selector(".quote", timeout=15_000) # timeout is in milliseconds ``` Some things worth knowing: - **Playwright auto-waits before actions.** Methods like `click()` and `fill()` automatically wait until the element is attached, visible, stable, enabled and able to receive events. The default timeout is 30 seconds. - **Prefer a specific wait to a general one.** `wait_for_selector()` waits for an element. `page.wait_for_function()` waits for any JavaScript condition you write, and we'll use it for scrolling below. - `wait_until` **controls when** `goto()` **returns.** The default, `"load"`, waits for the whole page and its resources. `"domcontentloaded"` returns sooner, when the HTML is parsed. `"networkidle"` waits for network quiet, but Playwright's documentation discourages relying on it: pages that poll or stream constantly never go quiet, and it slows things down. Combine `"domcontentloaded"` with a wait for the element you need. A practice page lets you see this in action. The `/js-delayed` page renders its content only after a delay, and a selector wait handles it without guessing how long that takes. ## Extracting Data: Locators or BeautifulSoup? Playwright can read the page directly with **locators**: ```python quotes = [] for card in page.locator(".quote").all(): quotes.append({ "text": card.locator(".text").inner_text(), "author": card.locator(".author").inner_text(), "tags": card.locator(".tag").all_inner_texts(), }) ``` Or you can pass `page.content()` to BeautifulSoup, as we did earlier: ```python def parse_quotes(html): soup = BeautifulSoup(html, "html.parser") return [ { "text": q.select_one(".text").get_text(strip=True), "author": q.select_one(".author").get_text(strip=True), "tags": [t.get_text(strip=True) for t in q.select(".tag")], } for q in soup.select(".quote") ] ``` Which should you pick? - **Use locators** when you're interacting with the page (clicking, typing, waiting for elements). - **Use BeautifulSoup** when you already have parsing code, or when you want to extract lots of data quickly. Each locator call is a round trip to the browser, while parsing one HTML string is a local operation. A common pattern is Playwright for getting the page into the right state, then BeautifulSoup for parsing it. ## Pagination On many sites, "Next" is a real link that loads a new page. The safest approach is to read the link's address and navigate to it directly: ```python import time from urllib.parse import urljoin from playwright.sync_api import sync_playwright START_URL = "http://quotes.toscrape.com/js/" with sync_playwright() as p: browser = p.chromium.launch() page = browser.new_page() url, all_quotes = START_URL, [] for _ in range(20): # safety cap on pages page.goto(url) page.wait_for_selector(".quote") all_quotes.extend(parse_quotes(page.content())) next_link = page.locator("li.next a") if next_link.count() == 0: # no Next link: last page break url = urljoin(page.url, next_link.get_attribute("href")) time.sleep(1.5) # be polite between pages browser.close() print(len(all_quotes)) ``` Two details prevent subtle bugs: - **Why not just click "Next"?** On pages that update in place (single-page apps), the old items stay on screen after a click until the new ones arrive, so `wait_for_selector(".quote")` can return immediately with the _previous_ page's quotes. Navigating to the link's URL avoids that race. - `count()` **doesn't wait.** It reports what's in the page right now. That's fine here because we've already waited for the quotes, but if a Next button renders later than the content, wait for it first. Also note that `get_attribute()` _does_ wait (up to the 30-second default), which is why we check `count()` before calling it. ## Infinite Scroll and "Load More" Buttons Some pages load more items as you scroll, or when you click a button, and the URL never changes. The trick is to **scroll (or click), then wait until more items exist**. This helper does the waiting: ```python from playwright.sync_api import TimeoutError as PlaywrightTimeoutError def wait_for_more(page, selector, previous_count, timeout=5_000): """Wait until more elements match `selector`. Returns False if none appear.""" try: page.wait_for_function( "([sel, n]) => document.querySelectorAll(sel).length > n", arg=[selector, previous_count], timeout=timeout, ) return True except PlaywrightTimeoutError: return False ``` Now loop until nothing new arrives: ```python import time from playwright.sync_api import sync_playwright def scrape_infinite_scroll(url, item_selector=".quote", max_scrolls=50): with sync_playwright() as p: browser = p.chromium.launch() page = browser.new_page() page.goto(url) page.wait_for_selector(item_selector) for _ in range(max_scrolls): # safety cap count = page.locator(item_selector).count() page.evaluate("window.scrollTo(0, document.body.scrollHeight)") if not wait_for_more(page, item_selector, count): break # nothing new loaded: we're done time.sleep(1.0) # be polite between loads html = page.content() browser.close() return parse_quotes(html) quotes = scrape_infinite_scroll("http://quotes.toscrape.com/scroll") print(len(quotes)) # 100 ``` For a "Load more" button, replace the `scrollTo` line with `page.click("button.load-more")` (using the site's real selector), and keep the same wait. Notice the termination rule: stop when a scroll produces no new items within a few seconds, and always keep a `max_scrolls` cap in case a site really does scroll forever. ## Capture the JSON the Page Loads Often the best of both worlds is to let the browser do the loading, then read the JSON it received instead of parsing the rendered HTML. You get clean structured data and you don't have to reverse-engineer tokens or headers: ```python from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch() page = browser.new_page() with page.expect_response(lambda r: "/api/quotes" in r.url) as response_info: page.goto("http://quotes.toscrape.com/scroll") data = response_info.value.json() print(data["page"], len(data["quotes"]), data["has_next"]) # 1 10 True browser.close() ``` `expect_response()` takes a function that decides which response you want, and the `with` block waits for it to arrive. To collect _every_ matching response while you scroll, register a listener: ```python captured = [] def on_response(response): if "/api/quotes" in response.url and response.ok: captured.append(response) page.on("response", on_response) # ...scroll as in the previous section, then: all_quotes = [q for r in captured for q in r.json()["quotes"]] ``` We store the response objects and decode them afterwards, which keeps the handler fast. ## Making It Faster and Lighter A browser downloads everything a page asks for: images, fonts, stylesheets, ads. For scraping you usually need only the HTML and the data. Playwright lets you intercept requests and skip the heavy ones: ```python BLOCKED_TYPES = {"image", "font", "media"} def block_heavy_resources(route): if route.request.resource_type in BLOCKED_TYPES: route.abort() else: route.continue_() page.route("**/*", block_heavy_resources) ``` Register the route before `goto()`. Don't block scripts, since they're what generates the content, and be cautious about stylesheets, because some sites rely on CSS for layout state. Blocking also means less load on the site you're visiting. More ways to keep a scraper lean: - **Reuse one browser.** Launching a browser is the expensive part. Open it once and create pages, or isolated **contexts**, from it. - **Use contexts for isolation.** Each `browser.new_context()` has its own cookies and storage, like a separate incognito profile. - **Keep concurrency low.** If you need several pages at once, the async API handles it. This example caps it at two at a time: ```python import asyncio from playwright.async_api import async_playwright async def scrape_page(browser, url, semaphore): async with semaphore: page = await browser.new_page() try: await page.goto(url) await page.wait_for_selector(".quote") return parse_quotes(await page.content()) finally: await page.close() async def main(urls): semaphore = asyncio.Semaphore(2) # at most two pages at once async with async_playwright() as p: browser = await p.chromium.launch() results = await asyncio.gather(*(scrape_page(browser, u, semaphore) for u in urls)) await browser.close() return [quote for page_quotes in results for quote in page_quotes] urls = [f"http://quotes.toscrape.com/js/page/{n}/" for n in range(1, 4)] quotes = asyncio.run(main(urls)) print(len(quotes)) # 30 ``` ## Being Honest About Who You Are By default, a headless browser's user agent contains `HeadlessChrome`, which plainly signals automation. Give your scraper an honest, descriptive user agent with contact details, just as you would with `requests`: ```python context = browser.new_context( user_agent="MyResearchBot/1.0 (+https://example.com/bot; contact@example.com)", viewport={"width": 1280, "height": 800}, ) page = context.new_page() ``` You'll find guides online about hiding the fact that a browser is automated, using "stealth" plugins and spoofed fingerprints. This post deliberately doesn't cover them. If a site works hard to stop automated visitors, that is the site telling you no, and working around it moves you toward the legal risks described in our guide to scraping law. ## Putting It Together Here's a complete script that checks `robots.txt`, identifies itself, skips heavy resources, follows pagination, parses with BeautifulSoup, and saves a CSV: ```python import csv import time from urllib.parse import urljoin, urlsplit from urllib.robotparser import RobotFileParser from bs4 import BeautifulSoup from playwright.sync_api import TimeoutError as PlaywrightTimeoutError from playwright.sync_api import sync_playwright START_URL = "http://quotes.toscrape.com/js/" USER_AGENT = "MyResearchBot/1.0 (+https://example.com/bot; contact@example.com)" BLOCKED_TYPES = {"image", "font", "media"} def allowed_by_robots(url): parts = urlsplit(url) rp = RobotFileParser(f"{parts.scheme}://{parts.netloc}/robots.txt") rp.read() return rp.can_fetch(USER_AGENT, url) def block_heavy_resources(route): if route.request.resource_type in BLOCKED_TYPES: route.abort() else: route.continue_() def parse_quotes(html): soup = BeautifulSoup(html, "html.parser") return [ { "text": q.select_one(".text").get_text(strip=True), "author": q.select_one(".author").get_text(strip=True), "tags": [t.get_text(strip=True) for t in q.select(".tag")], } for q in soup.select(".quote") ] def scrape_all(start_url, max_pages=20, delay=1.5): if not allowed_by_robots(start_url): raise PermissionError(f"robots.txt disallows {start_url}") quotes, url = [], start_url with sync_playwright() as p: browser = p.chromium.launch() context = browser.new_context(user_agent=USER_AGENT) page = context.new_page() page.route("**/*", block_heavy_resources) for number in range(1, max_pages + 1): print(f"Page {number}: {url}") try: page.goto(url, wait_until="domcontentloaded", timeout=20_000) page.wait_for_selector(".quote", timeout=10_000) except PlaywrightTimeoutError: print(" Timed out waiting for quotes, stopping.") break quotes.extend(parse_quotes(page.content())) next_link = page.locator("li.next a") if next_link.count() == 0: break url = urljoin(page.url, next_link.get_attribute("href")) time.sleep(delay) browser.close() return quotes def save_to_csv(rows, filename): with open(filename, "w", newline="", encoding="utf-8") as f: writer = csv.DictWriter(f, fieldnames=["text", "author", "tags"]) writer.writeheader() for row in rows: writer.writerow({**row, "tags": "; ".join(row["tags"])}) if __name__ == "__main__": data = scrape_all(START_URL) save_to_csv(data, "quotes.csv") print(f"Saved {len(data)} quotes") ``` Save it as `scraper.py` and run `python scraper.py`. The practice site has 100 quotes over ten pages, so with the polite delay it finishes in under half a minute. ## Responsible Scraping with a Real Browser A browser makes you look like a human visitor, which is exactly why you should be careful: - **Follow the rules.** Check `robots.txt` and the terms of service before you start, and prefer an official API when one exists. - **A browser is heavy for the site, too.** Every page load triggers dozens of requests. Add delays, keep concurrency low, and block resources you don't need. - **Don't log in with accounts you're not entitled to use,** and don't solve CAPTCHAs or work around bot protection to get at content a site is deliberately guarding. - **Don't hide your identity.** Use a descriptive user agent and contact details. - **Keep personal data out of it** unless you have a clear, lawful reason to collect it. Our guide "Is Web Scraping Legal?" covers the legal side in depth, and the `PoliteSession` class from it adds `robots.txt` caching and per-site rate limiting if you want a fuller version of the `allowed_by_robots` check above. ## Troubleshooting | Symptom | Likely cause | What to try | | --- | --- | --- | | Error saying the browser _executable doesn't exist_ | The browser build isn't installed | Run `playwright install chromium` | | Empty results, no error | You read `page.content()` before JavaScript finished | Add `wait_for_selector()` for the element you need | | `TimeoutError` waiting for a selector | Wrong selector, content never rendered, or a consent pop-up is covering the page | Take a screenshot (`page.screenshot(path="debug.png")`) and inspect it | | Works on your laptop, fails in Docker or CI | Missing system libraries | Install with `playwright install --with-deps chromium` | | Duplicate or missing items when paginating | You clicked "Next" and read the old page | Navigate to the link's URL, or wait for the item count to change | | Very slow scraping | Loading images, fonts and ads; one page at a time | Block heavy resources, reuse the browser, and add modest concurrency | To see what the browser is doing, launch it with `p.chromium.launch(headless=False, slow_mo=500)`. A visible window and a half-second pause after each action make bugs easy to spot. ## Wrapping Up Scraping a JavaScript-heavy site comes down to a short decision process: 1. **Check the raw HTML.** If the data is there, use `requests` and BeautifulSoup. 2. **Check the Network tab.** If the page loads JSON from an endpoint, call it directly. 3. **Otherwise, use Playwright.** Wait for specific elements rather than sleeping, navigate pagination by URL, scroll and wait for the item count to grow, and consider capturing the JSON the page loads. 4. **Be lean and honest.** Block heavy resources, keep concurrency low, identify yourself, and respect `robots.txt` and the site's terms. From here, natural next steps are saving results to a database, scheduling the scraper to run daily, and adding retries and logging so a flaky page doesn't stop a long job. Happy scraping, and please do it politely! * * * _Found this helpful? Share it with a fellow developer or leave your questions in the comments._