
This is general information, not legal advice. The author is not a lawyer. This post summarizes public court decisions and commentary as of November 2026. Pending cases and proposals are described as of the latest sources available when it was written (about November 10, 2026). Scraping law differs by country, changes quickly, and depends heavily on the facts. If real money, personal data, or a dispute is involved, talk to a qualified lawyer.
"Is web scraping legal?" is one of the most searched questions in the field, and the honest answer is unsatisfying: it depends. Scraping publicly available data is not automatically illegal, and it is not automatically safe either.
The good news is that "it depends" is not random. The risk comes from a handful of separate legal layers, and once you can name them you can make better decisions about what to scrape, how, and what to do with the results.
This guide walks through each layer with the landmark cases, explains what robots.txt really is, and ends with a practical checklist and a Python example of a scraper that behaves politely by default.
The Short Answer
Legality depends on four things:
What you collect (public facts, copyrighted text, personal data, biometrics)
How you collect it (logged out, logged in, bypassing blocks, hammering servers)
Where you and the site operate (US, EU, India and so on)
What you do afterwards (analyze, republish, sell, train a model)
"But it's public" answers only part of the first question. A page can be visible to anyone and still carry contract terms, copyright, privacy obligations, or technical protections.
The Five Layers of Scraping Risk
Layer | The question it asks | What's at stake |
|---|---|---|
1. Computer-access laws | Did you get past a barrier (login, paywall, IP block) you weren't authorized to cross? | Criminal and civil liability |
2. Contract (terms of service) | Did you agree to terms that ban scraping, and do they bind you? | Breach-of-contract damages, injunctions, account bans |
3. Copyright and database rights | Are you copying protected content or a protected database? | Infringement claims |
4. Privacy and data protection | Does the data identify people? | Regulatory fines, individual claims |
5. Anti-circumvention and torts | Did you defeat bot detection or put a strain on the site's servers? | Anti-circumvention claims, trespass-style claims |
Let's take them one at a time.
Layer 1: Computer-Access Laws
Most countries have laws against unauthorized access to computer systems. In the US the main one is the Computer Fraud and Abuse Act (CFAA). The question for scrapers is what "without authorization" means when a page is open to the whole internet.
Van Buren v. United States (2021)
In June 2021, the US Supreme Court decided Van Buren 6-3. The case was not about scraping, but it reshaped the CFAA. The Court used a "gates-up-or-down" idea: if a person is allowed through a gate to reach information, using that information for an improper purpose does not by itself violate the statute. The Court also warned that treating every breach of a computer-use policy as a crime would make millions of ordinary people criminals.
hiQ Labs v. LinkedIn (2019 to 2022)
The scraping case everyone cites. hiQ scraped public LinkedIn profiles, and LinkedIn tried to block it using the CFAA. At the preliminary-injunction stage, the Ninth Circuit sided with hiQ on the CFAA question, reasoning that the law's "without authorization" language doesn't reach a public website, and it reaffirmed that view in April 2022.
If the story ended there, it would be a green light. It didn't end there:
In November 2022, the district court ruled that the user-agreement provisions prohibiting scraping and fake profiles were enforceable in a breach-of-contract claim.
In December 2022, the parties settled with a consent judgment of $500,000 against hiQ, along with an injunction effectively barring future scraping of LinkedIn.
A common takeaway from the case: scraping public pages may not break the CFAA, but you can still lose on contract or tort claims.
What this means in practice
Think of the "gates" idea as a rough guide:
A public page, no login, no technical barrier: the gate is up. Computer-access laws are the least likely problem.
Behind a login, paywall, or IP block: the gate is down. Getting around it, for example with borrowed credentials, is where criminal exposure becomes a real risk.
If you've read our guide to rotating proxies in Python, note the point there too: using rotation to dodge a block the site deliberately put in place moves you from "reading public pages" toward "getting around a barrier."
Layer 2: Terms of Service (Contract Law)
For everyday scrapers, contract claims are often the most realistic risk. Many sites' terms forbid "automated means" of access. Whether those terms bind you turns on whether you actually agreed to them.
Meta v. Bright Data (2024)
In January 2024, Judge Edward Chen of the Northern District of California ruled for Bright Data on Meta's breach-of-contract claim. Meta's terms govern a person's "use" of Facebook and Instagram. Bright Data scraped public data while logged out, so the court found it was not "using" the services and wasn't bound the way an account holder would be. In February 2024, Meta dropped its one remaining claim and waived its right to appeal the ruling.
The ruling is narrow. It addressed logged-out scraping of public data. It doesn't give you cover for scraping while logged in, which is a very different situation because you accepted the terms to get an account.
Contract protection outside the US
In Ryanair v. PR Aviation (January 2015), the EU's Court of Justice ruled that the Database Directive doesn't apply to databases protected neither by copyright nor by the special database right. For those, a website owner can restrict scraping through its terms. Whether a particular database qualifies for protection is left to national courts.
What this means in practice
Never scrape while logged in to a service whose terms ban it. You've accepted the terms, which makes a breach claim much easier for the site to bring.
Read the terms. Search for "automated", "scrape", "crawl", "robot" and "bot."
Look for an official API or data license first. It's usually faster, more stable, and legally cleaner.
Keep a record of the terms and
robots.txtas they stood when you collected the data.
Layer 3: Copyright and Database Rights
Facts, such as a price or a date, generally aren't protected by copyright in the US. But the creative expression around them (articles, reviews, photos) is, and the selection and arrangement of a database can be too. In the EU there is also a dedicated database right.
Two practical consequences:
Collecting is different from republishing. Analyzing scraped data privately carries a different risk from copying a site's articles or images and publishing them.
Defenses are fact-specific. US "fair use" and similar exceptions elsewhere are judged case by case, which is why lawyers rarely give yes-or-no answers.
A note on AI training
If you're scraping to train machine-learning models, you are in a separate and fast-moving area. One example: in Bartz v. Anthropic, the judge ruled in June 2025 that training on the books was fair use, but that keeping a permanent library of pirated copies was not, and he left the piracy claims for trial. The case then settled for $1.5 billion, which received final court approval on July 20, 2026. Other AI copyright cases are still working through the courts, so don't treat any single ruling as settled law.
Layer 4: Privacy and Data Protection
This is the layer developers most often forget, and the one that can lead to the largest fines.
"Publicly available" does not mean "free to use." In the EU, the GDPR applies to personal data even when it is on a public web page. In September 2024, the Dutch data protection authority fined Clearview AI €30.5 million over a facial-recognition database built by scraping images from the internet. The authority said the company had no legal basis for collecting biometric data and hadn't informed the people involved.
If you're in India
India's Digital Personal Data Protection Act, 2023 is now moving into force. The government notified the DPDP Rules, 2025 on 13 November 2025, with a phased rollout: the main obligations (notice and consent, security safeguards, breach reporting, children's data and others) are due to apply from 13 May 2027. MeitY has floated shortening that window for some provisions (reported in January 2026), but no amendment had been notified in the sources available through mid-October 2026, so check the latest dates before planning around them.
The Act itself (section 3(c)(ii)) does not apply to personal data that has been made publicly available by the person it relates to, or by someone who is legally obliged under Indian law to publish it. That means "it's on the internet" is not enough on its own: data posted by a third party may not be covered. An IAPP analysis also notes that the government has said scraping public user data must still comply with the Act's consent and other obligations, which commentators see as in tension with the exemption. How it applies to scrapers is untested, so don't assume it protects you.
Other Indian laws can apply too, including the Information Technology Act, 2000 (sections 43 and 66 deal with unauthorized access to computer systems and the related criminal liability) and the Copyright Act, 1957.
What this means in practice
Avoid personal data unless you truly need it. If you can answer your question with aggregate or non-personal data, do.
Treat biometric, health, and children's data as off limits without expert advice.
If you must collect personal data, minimize it, secure it, set a retention period, and be able to explain your lawful basis.
Layer 5: Anti-Circumvention and Bot Detection
A newer front is whether bot-detection systems and CAPTCHAs count as digital locks under anti-circumvention law (section 1201 of the US DMCA). Courts reached different results in 2026:
In Reddit v. SerpApi (S.D.N.Y., July 31, 2026), the court let DMCA claims go forward at the pleading stage. It found that a bot-detection system can plausibly count as a measure that "effectively controls access," even though human users can read the same content. (This is a motion-to-dismiss ruling, so nothing has been proven.)
In Google v. SerpApi (N.D. Cal.), the court dismissed Google's anti-circumvention claims on July 20, 2026. Some were dismissed without leave to amend, because no copyrighted content was involved, and some with leave to amend, because Google hadn't shown it deployed its protection with the copyright owners' authorization. Google filed an amended complaint on August 10 with new authorization theories, but in early November 2026 (reported November 10) the court granted SerpApi's motion to dismiss that version too. Google was given until November 30 to try again, failing which the case would be dismissed with prejudice. The court did not rule that SerpApi's scraping is lawful, only that Google had not pleaded enough.
These are early rulings and both may change on amendment or appeal. The practical takeaway is simpler than the legal detail: defeating technical protections is the point where "reading public pages" starts to look like "breaking a lock." Even if a claim ultimately fails, being sued is expensive.
Separately, aggressive crawling that strains a site's servers has historically supported trespass-style claims. Being gentle is both polite and protective.
robots.txt: What It Is and What It Isn't
robots.txt is a plain text file at the root of a site (https://example.com/robots.txt) that tells crawlers which paths they should avoid. It follows the Robots Exclusion Protocol, standardized as RFC 9309 in 2022.
User-agent: *
Disallow: /private/
Disallow: /search
Crawl-delay: 5
User-agent: BadBot
Disallow: /Two things are true at once:
It is not a law, and it is not access control. The RFC itself says the rules are "not a form of access authorization" and are "not a substitute for valid content security measures."
You should follow it anyway. It is how site owners state their wishes, ignoring it makes you look like a bad actor if there is ever a dispute, and a site's terms of service may refer to it.
Some details worth knowing:
Crawl-delayisn't part of the standard. The RFC doesn't define it, though many sites use it and well-behaved crawlers honor it.If
robots.txtis unreachable (a server error or network failure), the RFC says to assume everything is disallowed.If it returns a 4xx error such as 404, the RFC says the crawler may access any resources. The example below is more cautious and treats 401 and 403 as "stay out."
Python's
urllib.robotparsercan read these rules, but it doesn't fetch the file with your user agent and doesn't enforce rate limits for you.
A scraper that behaves politely by default
This wrapper around requests does four things automatically: it identifies itself, checks robots.txt (once per site), honors Crawl-delay, and rate-limits requests to each site.
import time
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser
import requests
USER_AGENT = "MyResearchBot/1.0 (+https://example.com/bot; [email protected])"
class PoliteSession:
"""A requests wrapper that obeys robots.txt, identifies itself, and rate-limits."""
def __init__(self, user_agent=USER_AGENT, default_delay=2.0):
self.user_agent = user_agent
self.default_delay = default_delay # seconds between requests to one site
self.session = requests.Session()
self.session.headers["User-Agent"] = user_agent
self._robots = {} # origin -> parsed robots.txt
self._last_hit = {} # origin -> time of the last request
@staticmethod
def _origin(url):
parts = urlsplit(url)
return f"{parts.scheme}://{parts.netloc}"
def _rules(self, origin):
if origin not in self._robots:
rp = RobotFileParser()
try:
resp = self.session.get(f"{origin}/robots.txt", timeout=10)
except requests.RequestException:
rp.parse(["User-agent: *", "Disallow: /"]) # can't check, so stay out
else:
if resp.status_code in (401, 403) or resp.status_code >= 500:
rp.parse(["User-agent: *", "Disallow: /"])
elif resp.status_code >= 400: # e.g. 404: no rules published
rp.parse([])
else:
rp.parse(resp.text.splitlines())
self._robots[origin] = rp
return self._robots[origin]
def get(self, url, **kwargs):
origin = self._origin(url)
rules = self._rules(origin)
if not rules.can_fetch(self.user_agent, url):
raise PermissionError(f"robots.txt disallows {url}")
delay = rules.crawl_delay(self.user_agent) or self.default_delay
last = self._last_hit.get(origin)
if last is not None:
wait = last + delay - time.monotonic()
if wait > 0:
time.sleep(wait)
self._last_hit[origin] = time.monotonic()
kwargs.setdefault("timeout", 10)
return self.session.get(url, **kwargs)Using it:
bot = PoliteSession()
try:
response = bot.get("https://example.com/products")
print(response.status_code)
except PermissionError as exc:
print(f"Skipping: {exc}")A few notes:
Use a real contact address in the user agent. If your crawler causes trouble, the site owner can email you instead of blocking you or calling a lawyer.
The parsed rules are cached for the life of the object. For a long-running crawler, re-fetch
robots.txtperiodically.This doesn't make a scrape legal. It handles courtesy and one signal,
robots.txt. Terms of service, copyright, and privacy are still on you.
A Practical Checklist
Before you scrape, work through these:
Is there an API, dataset, or license? If yes, use it.
Is the data public and available without logging in? If you need an account, read the terms very carefully, because you're bound by them.
Have you read the terms of service and
robots.txt? Save a copy with the date.Does the data include personal information? If so, ask whether you really need it, and think about privacy law where you and the data subjects live.
Are you avoiding technical barriers? No bypassing logins, paywalls, CAPTCHAs, or IP bans.
Are you identifying yourself and going slowly? A clear user agent, contact details, delays, and off-peak hours.
What will you do with the data? Analysis, republishing, resale, and model training carry very different risk.
Can you explain your reasoning? Write down why you believe the collection is legitimate.
Red flags
Red flag | Why it matters |
|---|---|
You have to log in, or create fake accounts | Contract and computer-access exposure |
You're rotating IPs or solving CAPTCHAs to get around a block | Looks like circumventing a barrier |
The data identifies individuals, especially sensitive data | Privacy law, with potentially large fines |
You plan to republish the content wholesale | Copyright and database rights |
The site is small or fragile and you're sending heavy traffic | Server-strain (trespass-style) arguments |
The site has told you to stop | Continuing makes any later dispute much worse |
If you receive a cease-and-desist letter
Stop scraping that site right away, don't delete your records, and speak to a lawyer before replying. Ignoring a letter or answering it defensively tends to make matters worse.
Key Cases at a Glance
Date | Case | Why it matters |
|---|---|---|
Jan 2015 | Ryanair v. PR Aviation (EU Court of Justice) | Terms of service can restrict scraping of databases that lack IP protection |
Jun 2021 | Van Buren v. United States (US Supreme Court) | Narrowed the CFAA with the "gates-up-or-down" approach |
2019 to Apr 2022 | hiQ Labs v. LinkedIn (9th Cir.) | Scraping public pages likely isn't a CFAA violation |
Nov to Dec 2022 | hiQ v. LinkedIn (remand) | Scraping terms were enforceable; ended in a $500,000 consent judgment |
Jan 2024 | Meta v. Bright Data (N.D. Cal.) | Logged-out scraping of public data didn't breach Meta's terms |
Sep 2024 | Clearview AI fine (Dutch DPA) | Scraping can violate GDPR when it involves personal or biometric data |
Jun 2025 to Jul 2026 | Bartz v. Anthropic (N.D. Cal.) | AI-training ruling distinguishing training from pirated libraries; $1.5B settlement approved Jul 20, 2026 |
Nov 2025 | India DPDP Rules notified | Major obligations phase in from May 2027 (a shorter window has been floated, not notified, as of mid-October 2026) |
Jul to Nov 2026 | Reddit v. SerpApi (S.D.N.Y., Jul 31) and Google v. SerpApi (N.D. Cal., dismissed Jul 20 and again in Nov) | Different results, at an early stage, on whether bot detection is a "lock" under the DMCA; both cases are ongoing |
Wrapping Up
Scraping sits in a legal gray zone only in the sense that the answer depends on specifics, not that nobody knows the rules. The pattern from the cases is consistent:
Public, logged-out, non-personal data, collected gently, is the lowest-risk end of the spectrum.
Logging in, circumventing protections, collecting personal data, or republishing at scale pushes you up the risk ladder quickly.
Courtesy is protective. Honoring
robots.txt, identifying yourself, and going slowly won't make a scrape legal, but they make disputes less likely and easier to resolve.
When in doubt, use the API, ask the site owner, or ask a lawyer, in that order of cost.
Scrape responsibly.
Further Reading
hiQ v. LinkedIn: landmark scraping suit provides guidance (Morgan Lewis)
Van Buren: the Supreme Court limits the scope of the CFAA (Morrison Foerster)
Court dismisses Google's DMCA claims against SerpApi (Search Engine Journal)
Google loses another round in legal battle against SerpApi (Search Engine Roundtable)
ECJ clarifies Database Directive scope in screen scraping case (Ryanair)
Clearview AI hit with its largest GDPR fine yet (TechCrunch)
Judge approves record $1.5 billion AI copyright settlement (JURIST)
India's DPDP Act brought into force (Hogan Lovells Cadwalader)
Scraping public data in India: innovation enabler or privacy threat? (IAPP)
This article is general information, not legal advice. Laws and court decisions change, and the right answer depends on your facts and jurisdiction. Talk to a qualified lawyer before relying on any of it.
Comments(0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!