Scraping

Scraping JavaScript-Heavy Websites with Playwright in Python

Learn to scrape JavaScript-heavy sites in Python with Playwright: smart waiting, pagination, infinite scroll, JSON capture, and BeautifulSoup parsing.

·
10 min read
·
AI-assisted

AI tooling disclosure

Generative AI materially contributed to this article.

Human review confirmed by the author.

Scraping JavaScript-Heavy Websites with Playwright in Python

You write a clean requests and BeautifulSoup scraper, point it at a page that clearly shows a list of items in your browser, and get back... nothing. An empty list, or a page full of placeholders.

That's the signature of a JavaScript-rendered site. The server sends a nearly empty HTML shell, and JavaScript running in your browser fills in the content afterwards. requests never runs that JavaScript, so it never sees the data.

In this guide you'll learn how to diagnose the problem, decide whether you need a browser at all, and use Playwright to scrape these sites, including pagination, infinite scroll, and capturing the JSON the page loads. We'll finish with a complete script that saves results to CSV.

Before you scrape: check the site's terms of service and robots.txt, and read our guide "Is Web Scraping Legal?" first if you haven't. A real browser makes it easy to do things a site's owner may not welcome, so the responsible-use section near the end matters even more here.


Why requests and BeautifulSoup Come Back Empty

Two views of the same page are easy to confuse:

  • View Source shows exactly what the server sent. That's what requests gets.

  • Inspect Element shows the live page after JavaScript has run.

If the data appears in Inspect but not in View Source, JavaScript is building it. You can confirm with a quick test against the practice site quotes.toscrape.com, which renders its quotes with JavaScript:

import requests
from bs4 import BeautifulSoup

html = requests.get("http://quotes.toscrape.com/js/", timeout=10).text
soup = BeautifulSoup(html, "html.parser")

print(len(soup.select(".quote")))     # 0: the quotes aren't in the raw HTML

Zero results from a selector you can clearly see in your browser's Inspect panel is the telltale sign.

Do You Need a Browser at All?

A real browser is the heaviest tool in the box. Before you reach for it, check where the data actually comes from:

Where the data lives

Best tool

In the raw HTML (View Source)

requests + BeautifulSoup

Loaded from a JSON endpoint you can see in the browser's Network tab

Call that endpoint directly with requests

Only exists after JavaScript runs, or needs scrolling, clicking, or other interaction

Playwright

The second row is worth a look every time, because it's often the cleanest solution. Open your browser's developer tools, switch to the Network tab, reload the page, and filter by Fetch/XHR. On our practice site the infinite-scroll page loads its quotes from a JSON endpoint:

import requests

response = requests.get(
    "http://quotes.toscrape.com/api/quotes", params={"page": 1}, timeout=10
)
data = response.json()

print(len(data["quotes"]), data["has_next"])    # 10 True

No browser needed. But sometimes the endpoint needs headers or tokens that the page generates, or the content only appears after a series of clicks. That's where Playwright earns its keep.

Setting Up Playwright

Playwright is a browser automation library from Microsoft. It drives real Chromium, Firefox, or WebKit browsers, so JavaScript runs exactly as it would for a human visitor.

python -m venv venv
source venv/bin/activate        # On Windows: venv\Scripts\activate
pip install playwright beautifulsoup4 requests
playwright install chromium

Two notes:

  • playwright install chromium downloads a browser build that Playwright manages itself. Without that step, the first launch fails.

  • On a bare Linux server or in CI, use playwright install --with-deps chromium instead, which also installs the operating-system libraries the browser needs.

Your First Playwright Scrape

from bs4 import BeautifulSoup
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()           # headless by default: no window opens
    page = browser.new_page()
    page.goto("http://quotes.toscrape.com/js/")
    page.wait_for_selector(".quote")        # wait until JavaScript has rendered the quotes
    html = page.content()                   # the page *after* JavaScript ran
    browser.close()

soup = BeautifulSoup(html, "html.parser")
print(len(soup.select(".quote")))           # 10

What happened:

  1. Playwright launched a real browser in the background (headless means no visible window).

  2. page.goto() loaded the page, and the browser ran its JavaScript.

  3. page.wait_for_selector(".quote") paused until at least one quote existed in the page.

  4. page.content() returned the rendered HTML, which we handed to BeautifulSoup. The parsing code is the same as in any other scraper.

Waiting the Right Way

Timing is the heart of scraping dynamic sites. Grab the HTML too early and you get an empty page, which is the most common beginner bug.

Avoid fixed sleeps. time.sleep(5) is either too short (empty results) or too long (slow scraping). Wait for the thing you actually need:

# Fragile: guesses how long rendering takes
page.goto(url)
time.sleep(5)

# Better: wait for the element you need
page.goto(url)
page.wait_for_selector(".quote", timeout=15_000)    # timeout is in milliseconds

Some things worth knowing:

  • Playwright auto-waits before actions. Methods like click() and fill() automatically wait until the element is attached, visible, stable, enabled and able to receive events. The default timeout is 30 seconds.

  • Prefer a specific wait to a general one. wait_for_selector() waits for an element. page.wait_for_function() waits for any JavaScript condition you write, and we'll use it for scrolling below.

  • wait_until controls when goto() returns. The default, "load", waits for the whole page and its resources. "domcontentloaded" returns sooner, when the HTML is parsed. "networkidle" waits for network quiet, but Playwright's documentation discourages relying on it: pages that poll or stream constantly never go quiet, and it slows things down. Combine "domcontentloaded" with a wait for the element you need.

A practice page lets you see this in action. The /js-delayed page renders its content only after a delay, and a selector wait handles it without guessing how long that takes.

Extracting Data: Locators or BeautifulSoup?

Playwright can read the page directly with locators:

quotes = []
for card in page.locator(".quote").all():
    quotes.append({
        "text": card.locator(".text").inner_text(),
        "author": card.locator(".author").inner_text(),
        "tags": card.locator(".tag").all_inner_texts(),
    })

Or you can pass page.content() to BeautifulSoup, as we did earlier:

def parse_quotes(html):
    soup = BeautifulSoup(html, "html.parser")
    return [
        {
            "text": q.select_one(".text").get_text(strip=True),
            "author": q.select_one(".author").get_text(strip=True),
            "tags": [t.get_text(strip=True) for t in q.select(".tag")],
        }
        for q in soup.select(".quote")
    ]

Which should you pick?

  • Use locators when you're interacting with the page (clicking, typing, waiting for elements).

  • Use BeautifulSoup when you already have parsing code, or when you want to extract lots of data quickly. Each locator call is a round trip to the browser, while parsing one HTML string is a local operation.

A common pattern is Playwright for getting the page into the right state, then BeautifulSoup for parsing it.

Pagination

On many sites, "Next" is a real link that loads a new page. The safest approach is to read the link's address and navigate to it directly:

import time
from urllib.parse import urljoin

from playwright.sync_api import sync_playwright

START_URL = "http://quotes.toscrape.com/js/"

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    url, all_quotes = START_URL, []

    for _ in range(20):                                  # safety cap on pages
        page.goto(url)
        page.wait_for_selector(".quote")
        all_quotes.extend(parse_quotes(page.content()))

        next_link = page.locator("li.next a")
        if next_link.count() == 0:                       # no Next link: last page
            break
        url = urljoin(page.url, next_link.get_attribute("href"))
        time.sleep(1.5)                                  # be polite between pages

    browser.close()

print(len(all_quotes))

Two details prevent subtle bugs:

  • Why not just click "Next"? On pages that update in place (single-page apps), the old items stay on screen after a click until the new ones arrive, so wait_for_selector(".quote") can return immediately with the previous page's quotes. Navigating to the link's URL avoids that race.

  • count() doesn't wait. It reports what's in the page right now. That's fine here because we've already waited for the quotes, but if a Next button renders later than the content, wait for it first. Also note that get_attribute() does wait (up to the 30-second default), which is why we check count() before calling it.

Infinite Scroll and "Load More" Buttons

Some pages load more items as you scroll, or when you click a button, and the URL never changes. The trick is to scroll (or click), then wait until more items exist. This helper does the waiting:

from playwright.sync_api import TimeoutError as PlaywrightTimeoutError


def wait_for_more(page, selector, previous_count, timeout=5_000):
    """Wait until more elements match `selector`. Returns False if none appear."""
    try:
        page.wait_for_function(
            "([sel, n]) => document.querySelectorAll(sel).length > n",
            arg=[selector, previous_count],
            timeout=timeout,
        )
        return True
    except PlaywrightTimeoutError:
        return False

Now loop until nothing new arrives:

import time

from playwright.sync_api import sync_playwright


def scrape_infinite_scroll(url, item_selector=".quote", max_scrolls=50):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url)
        page.wait_for_selector(item_selector)

        for _ in range(max_scrolls):                      # safety cap
            count = page.locator(item_selector).count()
            page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
            if not wait_for_more(page, item_selector, count):
                break                                     # nothing new loaded: we're done
            time.sleep(1.0)                               # be polite between loads

        html = page.content()
        browser.close()
    return parse_quotes(html)


quotes = scrape_infinite_scroll("http://quotes.toscrape.com/scroll")
print(len(quotes))                                        # 100

For a "Load more" button, replace the scrollTo line with page.click("button.load-more") (using the site's real selector), and keep the same wait. Notice the termination rule: stop when a scroll produces no new items within a few seconds, and always keep a max_scrolls cap in case a site really does scroll forever.

Capture the JSON the Page Loads

Often the best of both worlds is to let the browser do the loading, then read the JSON it received instead of parsing the rendered HTML. You get clean structured data and you don't have to reverse-engineer tokens or headers:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()

    with page.expect_response(lambda r: "/api/quotes" in r.url) as response_info:
        page.goto("http://quotes.toscrape.com/scroll")

    data = response_info.value.json()
    print(data["page"], len(data["quotes"]), data["has_next"])    # 1 10 True
    browser.close()

expect_response() takes a function that decides which response you want, and the with block waits for it to arrive. To collect every matching response while you scroll, register a listener:

captured = []


def on_response(response):
    if "/api/quotes" in response.url and response.ok:
        captured.append(response)


page.on("response", on_response)
# ...scroll as in the previous section, then:
all_quotes = [q for r in captured for q in r.json()["quotes"]]

We store the response objects and decode them afterwards, which keeps the handler fast.

Making It Faster and Lighter

A browser downloads everything a page asks for: images, fonts, stylesheets, ads. For scraping you usually need only the HTML and the data. Playwright lets you intercept requests and skip the heavy ones:

BLOCKED_TYPES = {"image", "font", "media"}


def block_heavy_resources(route):
    if route.request.resource_type in BLOCKED_TYPES:
        route.abort()
    else:
        route.continue_()


page.route("**/*", block_heavy_resources)

Register the route before goto(). Don't block scripts, since they're what generates the content, and be cautious about stylesheets, because some sites rely on CSS for layout state. Blocking also means less load on the site you're visiting.

More ways to keep a scraper lean:

  • Reuse one browser. Launching a browser is the expensive part. Open it once and create pages, or isolated contexts, from it.

  • Use contexts for isolation. Each browser.new_context() has its own cookies and storage, like a separate incognito profile.

  • Keep concurrency low. If you need several pages at once, the async API handles it. This example caps it at two at a time:

import asyncio

from playwright.async_api import async_playwright


async def scrape_page(browser, url, semaphore):
    async with semaphore:
        page = await browser.new_page()
        try:
            await page.goto(url)
            await page.wait_for_selector(".quote")
            return parse_quotes(await page.content())
        finally:
            await page.close()


async def main(urls):
    semaphore = asyncio.Semaphore(2)                  # at most two pages at once
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        results = await asyncio.gather(*(scrape_page(browser, u, semaphore) for u in urls))
        await browser.close()
    return [quote for page_quotes in results for quote in page_quotes]


urls = [f"http://quotes.toscrape.com/js/page/{n}/" for n in range(1, 4)]
quotes = asyncio.run(main(urls))
print(len(quotes))                                    # 30

Being Honest About Who You Are

By default, a headless browser's user agent contains HeadlessChrome, which plainly signals automation. Give your scraper an honest, descriptive user agent with contact details, just as you would with requests:

context = browser.new_context(
    user_agent="MyResearchBot/1.0 (+https://example.com/bot; [email protected])",
    viewport={"width": 1280, "height": 800},
)
page = context.new_page()

You'll find guides online about hiding the fact that a browser is automated, using "stealth" plugins and spoofed fingerprints. This post deliberately doesn't cover them. If a site works hard to stop automated visitors, that is the site telling you no, and working around it moves you toward the legal risks described in our guide to scraping law.

Putting It Together

Here's a complete script that checks robots.txt, identifies itself, skips heavy resources, follows pagination, parses with BeautifulSoup, and saves a CSV:

import csv
import time
from urllib.parse import urljoin, urlsplit
from urllib.robotparser import RobotFileParser

from bs4 import BeautifulSoup
from playwright.sync_api import TimeoutError as PlaywrightTimeoutError
from playwright.sync_api import sync_playwright

START_URL = "http://quotes.toscrape.com/js/"
USER_AGENT = "MyResearchBot/1.0 (+https://example.com/bot; [email protected])"
BLOCKED_TYPES = {"image", "font", "media"}


def allowed_by_robots(url):
    parts = urlsplit(url)
    rp = RobotFileParser(f"{parts.scheme}://{parts.netloc}/robots.txt")
    rp.read()
    return rp.can_fetch(USER_AGENT, url)


def block_heavy_resources(route):
    if route.request.resource_type in BLOCKED_TYPES:
        route.abort()
    else:
        route.continue_()


def parse_quotes(html):
    soup = BeautifulSoup(html, "html.parser")
    return [
        {
            "text": q.select_one(".text").get_text(strip=True),
            "author": q.select_one(".author").get_text(strip=True),
            "tags": [t.get_text(strip=True) for t in q.select(".tag")],
        }
        for q in soup.select(".quote")
    ]


def scrape_all(start_url, max_pages=20, delay=1.5):
    if not allowed_by_robots(start_url):
        raise PermissionError(f"robots.txt disallows {start_url}")

    quotes, url = [], start_url
    with sync_playwright() as p:
        browser = p.chromium.launch()
        context = browser.new_context(user_agent=USER_AGENT)
        page = context.new_page()
        page.route("**/*", block_heavy_resources)

        for number in range(1, max_pages + 1):
            print(f"Page {number}: {url}")
            try:
                page.goto(url, wait_until="domcontentloaded", timeout=20_000)
                page.wait_for_selector(".quote", timeout=10_000)
            except PlaywrightTimeoutError:
                print("  Timed out waiting for quotes, stopping.")
                break

            quotes.extend(parse_quotes(page.content()))

            next_link = page.locator("li.next a")
            if next_link.count() == 0:
                break
            url = urljoin(page.url, next_link.get_attribute("href"))
            time.sleep(delay)

        browser.close()
    return quotes


def save_to_csv(rows, filename):
    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["text", "author", "tags"])
        writer.writeheader()
        for row in rows:
            writer.writerow({**row, "tags": "; ".join(row["tags"])})


if __name__ == "__main__":
    data = scrape_all(START_URL)
    save_to_csv(data, "quotes.csv")
    print(f"Saved {len(data)} quotes")

Save it as scraper.py and run python scraper.py. The practice site has 100 quotes over ten pages, so with the polite delay it finishes in under half a minute.

Responsible Scraping with a Real Browser

A browser makes you look like a human visitor, which is exactly why you should be careful:

  • Follow the rules. Check robots.txt and the terms of service before you start, and prefer an official API when one exists.

  • A browser is heavy for the site, too. Every page load triggers dozens of requests. Add delays, keep concurrency low, and block resources you don't need.

  • Don't log in with accounts you're not entitled to use, and don't solve CAPTCHAs or work around bot protection to get at content a site is deliberately guarding.

  • Don't hide your identity. Use a descriptive user agent and contact details.

  • Keep personal data out of it unless you have a clear, lawful reason to collect it.

Our guide "Is Web Scraping Legal?" covers the legal side in depth, and the PoliteSession class from it adds robots.txt caching and per-site rate limiting if you want a fuller version of the allowed_by_robots check above.

Troubleshooting

Symptom

Likely cause

What to try

Error saying the browser executable doesn't exist

The browser build isn't installed

Run playwright install chromium

Empty results, no error

You read page.content() before JavaScript finished

Add wait_for_selector() for the element you need

TimeoutError waiting for a selector

Wrong selector, content never rendered, or a consent pop-up is covering the page

Take a screenshot (page.screenshot(path="debug.png")) and inspect it

Works on your laptop, fails in Docker or CI

Missing system libraries

Install with playwright install --with-deps chromium

Duplicate or missing items when paginating

You clicked "Next" and read the old page

Navigate to the link's URL, or wait for the item count to change

Very slow scraping

Loading images, fonts and ads; one page at a time

Block heavy resources, reuse the browser, and add modest concurrency

To see what the browser is doing, launch it with p.chromium.launch(headless=False, slow_mo=500). A visible window and a half-second pause after each action make bugs easy to spot.

Wrapping Up

Scraping a JavaScript-heavy site comes down to a short decision process:

  1. Check the raw HTML. If the data is there, use requests and BeautifulSoup.

  2. Check the Network tab. If the page loads JSON from an endpoint, call it directly.

  3. Otherwise, use Playwright. Wait for specific elements rather than sleeping, navigate pagination by URL, scroll and wait for the item count to grow, and consider capturing the JSON the page loads.

  4. Be lean and honest. Block heavy resources, keep concurrency low, identify yourself, and respect robots.txt and the site's terms.

From here, natural next steps are saving results to a database, scheduling the scraper to run daily, and adding retries and logging so a flaky page doesn't stop a long job.

Happy scraping, and please do it politely!


Found this helpful? Share it with a fellow developer or leave your questions in the comments.

Comments(0)

Join the discussion by logging into your account.

No comments yet. Be the first to comment!

Bhavya Arora

Passionate developer sharing knowledge about modern web technologies and best practices.

Subscribe to Bhavya Arora's Newsletter

Direct email dispatches when new stories are published. Zero algorithms.