LinkedIn is one of the largest publicly accessible sources of structured professional data on the web. Public profile and company pages can expose useful professional metadata, while the jobs surface provides a large, searchable source of listings. Getting that data programmatically is harder: LinkedIn uses significant anti-abuse defenses, its official APIs expose only a subset of member and platform data to most developers, and the commercial scraping landscape changed significantly after Proxycurl shut down in 2025. This guide focuses on techniques that are useful in August 2026, while treating undocumented website behavior as implementation detail rather than a stable API.
Quick Start
One self-contained snippet — no config, no classes. If this returns data, your environment is working. Read the rest of the guide for rate limits, proxies, and production patterns.
# pip install curl-cffi parsel
import asyncio, json
from curl_cffi.requests import AsyncSession
from parsel import Selector
async def one_profile(url: str) -> dict:
async with AsyncSession(impersonate="chrome") as s:
r = await s.get(
url,
headers={"Accept-Language": "en-US,en;q=0.9"},
timeout=20,
)
for raw in Selector(r.text).xpath("//script[@type='application/ld+json']/text()").getall():
try:
data = json.loads(raw)
except json.JSONDecodeError:
continue
if isinstance(data, dict):
nodes = data.get("@graph", [data])
elif isinstance(data, list):
nodes = data
else:
nodes = []
for node in nodes:
if not isinstance(node, dict):
continue
node_type = node.get("@type")
if node_type == "Person" or (isinstance(node_type, list) and "Person" in node_type):
return {
"name": node.get("name"),
"headline": node.get("description"),
}
return {}
print(json.dumps(asyncio.run(one_profile("https://www.linkedin.com/in/williamhgates")), indent=2))
Legal Takeaways
The Ninth Circuit's April 2022 hiQ Labs v. LinkedIn decision is important for scraping publicly accessible data, but it is not a blanket authorization to scrape LinkedIn. The court's CFAA analysis distinguished public websites from access that requires authentication. Other claims, including contract claims under LinkedIn's User Agreement, remain separate. See the Ninth Circuit opinion.
The hiQ Labs settlement in December 2022 included a $500,000 judgment against hiQ for contract violations, per Morgan Lewis's summary. The hiQ case resolved both questions differently: CFAA in hiQ's favor, contract in LinkedIn's favor.
LinkedIn announced legal proceedings against Proxycurl on January 24, 2025, saying the action was intended to enforce its User Agreement against unauthorized scraping and fake accounts. LinkedIn later announced on July 28, 2025 that the lawsuit had been resolved. LinkedIn's announcement · LinkedIn's resolution announcement
Anyone reselling LinkedIn data or running a commercial-scale pipeline should talk to a lawyer before shipping.
What the Official API Actually Provides {#official-api}
LinkedIn's current API documentation lists three open permissions available to all developers without special approval: profile, email, and w_member_social. Many other products and permissions require explicit approval. For developers who need arbitrary third-party public profile data at scale, the standard self-service API does not provide an equivalent dataset. LinkedIn API access documentation
Where LinkedIn Stores Its Data
Public profiles and company pages can expose structured data in <script type="application/ld+json"> tags in the initial HTML response. The exact fields vary by page, but the data can provide useful identity, organization, and other metadata without requiring JavaScript execution. Parsing structured data is often preferable to relying entirely on presentation-oriented CSS classes, but it should still be treated as observed website behavior rather than a guaranteed schema.
One page can contain multiple ld+json blocks. The parse_ld_json helper below iterates all of them — not just the first — which matters because the Person or Organization node isn't always in the first block.
Job search results use a different pattern. The public jobs frontend currently uses an undocumented endpoint named jobs-guest/jobs/api/seeMoreJobPostings/search to request additional result pages. The response is HTML rather than a conventional JSON API. Treat this as an implementation detail: LinkedIn can change the endpoint, parameters, pagination behavior, or markup without notice.
Install: Why curl_cffi Instead of httpx
Some anti-bot systems inspect TLS and HTTP fingerprints in addition to higher-level request behavior. curl_cffi supports browser impersonation, which can make an HTTP client's TLS/HTTP fingerprint resemble a supported browser profile. That does not make a scraper invisible or establish how LinkedIn internally scores requests. See the curl_cffi documentation.
pip install curl-cffi parsel httpx
Keeping the fingerprint current. Every code sample below uses impersonate="chrome" (no version suffix). The library resolves this alias to its latest available Chrome profile — currently chrome146 as of August 2026 — rather than pinning you to a specific version. Since curl_cffi v0.15.1, you can also run curl-cffi update after install to pull the newest fingerprint definitions without a full pip upgrade. Pinning a version like impersonate="chrome146" is useful when you need reproducible fingerprints for debugging; otherwise, use the alias.
Shared Helpers
These two functions are used across every scraper below. get_html handles LinkedIn's 999 soft-block with exponential back-off. parse_ld_json iterates all ld+json blocks on the page and returns the first node matching the requested Schema.org type.
# helpers.py
import asyncio
import json
import random
from typing import Any
from curl_cffi.requests import AsyncSession
from parsel import Selector
HEADERS = {
"Accept-Language": "en-US,en;q=0.9",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}
async def get_html(session: AsyncSession, url: str, retries: int = 3) -> str:
"""Fetch a URL and retry selected transient LinkedIn responses."""
for attempt in range(retries):
try:
r = await session.get(url, headers=HEADERS, timeout=20)
except Exception:
if attempt == retries - 1:
return ""
await asyncio.sleep(2 ** attempt + random.uniform(0, 1))
continue
if r.status_code not in (302, 429, 999):
return r.text
await asyncio.sleep(2 ** attempt + random.uniform(0, 1))
return ""
def parse_ld_json(html: str, node_type: str) -> dict[str, Any]:
"""Return the first JSON-LD node matching node_type."""
selector = Selector(html)
for raw in selector.xpath(
"//script[@type='application/ld+json']/text()"
).getall():
try:
data = json.loads(raw)
except json.JSONDecodeError:
continue
if isinstance(data, dict):
nodes = data.get("@graph", [data])
elif isinstance(data, list):
nodes = data
else:
nodes = []
for node in nodes:
if not isinstance(node, dict):
continue
node_type_value = node.get("@type")
if node_type_value == node_type or (
isinstance(node_type_value, list)
and node_type in node_type_value
):
return node
return {}
Scraping Profiles
scrape_profiles wraps the helpers with a concurrency semaphore and an inter-request delay. The defaults are conservative starting points, not LinkedIn limits. The right values depend on the page type, network path, response behavior, and your workload. Treat rising 999 responses as a signal to reduce load and investigate rather than as a fixed quota.
# profiles.py
import asyncio
import json
import random
from curl_cffi.requests import AsyncSession
from helpers import get_html, parse_ld_json
async def scrape_profiles(
urls: list[str],
proxy: str | None = None, # "http://user:pass@host:port"
concurrency: int = 3,
delay: float = 2.5,
) -> list[dict]:
sem = asyncio.Semaphore(concurrency)
# impersonate="chrome" tracks the latest available Chrome fingerprint.
# Pin a version (e.g. "chrome146") only if you need reproducible fingerprints.
kw: dict = {"impersonate": "chrome"}
if proxy:
kw["proxies"] = {"https": proxy, "http": proxy}
async def fetch(url: str) -> dict:
async with sem:
async with AsyncSession(**kw) as s:
html = await get_html(s, url)
await asyncio.sleep(delay + random.uniform(0, 1))
if not html:
return {"url": url, "error": "blocked"}
node = parse_ld_json(html, "Person")
return {
"name": node.get("name"),
"headline": node.get("description"),
"employer": (node.get("worksFor") or [{}])[0].get("name"),
"url": url,
}
return list(await asyncio.gather(*[fetch(u) for u in urls]))
if __name__ == "__main__":
urls = [
"https://www.linkedin.com/in/williamhgates",
"https://www.linkedin.com/in/satyanadella",
]
print(json.dumps(asyncio.run(scrape_profiles(urls)), indent=2))
Adding a Residential Proxy
A proxy can change the network path and therefore the observed IP reputation, but it is not a universal requirement and it does not solve every form of anti-abuse control. Keep proxy support configurable so you can measure whether it actually improves your workload.
# Residential proxy from any provider (Bright Data, Smartproxy, Oxylabs, etc.)
PROXY = "http://username:[email protected]:8080"
results = asyncio.run(
scrape_profiles(
urls=["https://www.linkedin.com/in/williamhgates"],
proxy=PROXY,
concurrency=2, # lower concurrency when rotating proxies
delay=3.0,
)
)Scraping Company Pages
Company pages can use the same ld+json approach. An Organization node may expose fields such as name and description, with other attributes depending on the page. A secondary XPath pass can pick up additional fields from the DOM, but those selectors are inherently fragile. Treat structured data as a useful first layer and DOM extraction as best-effort.
# companies.py
import asyncio
import json
from curl_cffi.requests import AsyncSession
from parsel import Selector
from helpers import get_html, parse_ld_json
async def scrape_company(url: str, proxy: str | None = None) -> dict:
# impersonate="chrome" tracks the latest available Chrome fingerprint.
kw: dict = {"impersonate": "chrome"}
if proxy:
kw["proxies"] = {"https": proxy, "http": proxy}
async with AsyncSession(**kw) as s:
html = await get_html(s, url)
if not html:
return {}
base = parse_ld_json(html, "Organization")
# Best-effort: pick up extra About fields from the DOM.
# These selectors break when LinkedIn rotates its markup — treat as supplemental.
sel = Selector(html)
extra: dict = {}
for row in sel.xpath("//dl[contains(@class,'org-about')]"):
key = row.xpath("dt/text()").get("").strip()
val = " ".join(row.xpath("dd//text()").getall()).strip()
if key and val:
extra[key] = val
employees = base.get("numberOfEmployees")
employee_count = None
if isinstance(employees, dict):
employee_count = employees.get("value")
return {
"name": base.get("name"),
"description": base.get("description"),
"employees": employee_count,
"industry": base.get("industry"),
"website": base.get("url"),
**extra,
}
if __name__ == "__main__":
print(json.dumps(
asyncio.run(scrape_company("https://www.linkedin.com/company/microsoft")),
indent=2,
))Scraping Jobs via the Hidden API
The jobs example uses httpx because it does not depend on browser TLS impersonation. The pattern is straightforward: fetch the first search page, parse the initial cards and count, then request additional pages from the undocumented seeMoreJobPostings endpoint. The sleep between pages is a conservative load-management choice, not a documented LinkedIn requirement.
Note on parse_job_cards selectors. The XPath selectors below target LinkedIn's DOM structure as of August 2026. Like all DOM-based selectors, they'll break when LinkedIn updates its markup — this has happened several times in the past 18 months. If title fields come back empty, inspect a raw response and update the selectors. The ld+json approach used for profiles doesn't apply here because job cards on the paginated API response don't embed Schema.org nodes.
# jobs.py
import asyncio
import json
import math
import httpx
from parsel import Selector
from urllib.parse import urlencode
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/131.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
API_BASE = "https://www.linkedin.com/jobs-guest/jobs/api/seeMoreJobPostings/search?"
def parse_job_cards(html: str) -> list[dict]:
sel = Selector(html)
jobs = []
for li in sel.xpath("//li"):
title = li.xpath(".//div/a/span/text()").get("").strip()
company = li.xpath(".//h4/a/text()").get("").strip()
location= li.xpath(".//span[@class='job-search-card__location']/text()").get("").strip()
posted = li.xpath(".//time/@datetime").get()
url = li.xpath(".//a/@href").get("").split("?")[0]
if title:
jobs.append({
"title": title, "company": company,
"location": location, "posted": posted, "url": url,
})
return jobs
async def scrape_job_search(
keyword: str, location: str = "", max_results: int = 100
) -> list[dict]:
# urlencode handles query-string encoding internally — don't pre-encode
# values with quote_plus or you'll double-encode spaces as %2B.
qs = urlencode({"keywords": keyword, "location": location})
async with httpx.AsyncClient(headers=HEADERS, follow_redirects=True) as client:
first = await client.get(f"https://www.linkedin.com/jobs/search?{qs}")
count_raw = (
Selector(first.text)
.xpath("//span[contains(@class,'job-count')]/text()")
.get("0")
)
total = min(int(count_raw.replace(",", "").replace("+", "") or 0), max_results)
jobs = parse_job_cards(first.text)
for i in range(1, math.ceil((total - len(jobs)) / 25) + 1):
r = await client.get(API_BASE + qs + f"&start={i * 25}")
jobs.extend(parse_job_cards(r.text))
await asyncio.sleep(1.5)
return jobs
if __name__ == "__main__":
results = asyncio.run(
scrape_job_search("Python Developer", "United States", max_results=75)
)
print(json.dumps(results[:3], indent=2))
What Breaks at Scale
The impersonate="chrome" flag changes the TLS/HTTP fingerprint presented by the client. It is only one part of request behavior. Concurrency, timing, session state, IP/network reputation, and page type can all affect reliability, and LinkedIn can change its defenses over time.
Do not build production capacity around a fixed CAPTCHA or request threshold. Measure your own success and failure rates, record response classes, and apply back-pressure when the failure rate increases.
The Library Option
If you need authenticated data — connection lists, full post engagement, private contact info — the maintained open-source option is joeyism/linkedin_scraper v3.1.2 (April 2026), per ScrapFly's August 2026 review. Install it with pip install linkedin-scraper (not linkedin-scraper-patchright, which is a separate unrelated package).
⚠️ Check the license before shipping. The repository's
LICENSEfile is GPL-3.0. The README has contained conflicting license information, so inspect the actual license and the exact revision you plan to depend on rather than relying on the README alone. Whether and how GPL obligations apply to your particular product depends on how you use and distribute the code; get legal advice for a commercial product if necessary. Also test the current release against your target pages before committing to it.
FAQ
Does scraping LinkedIn violate the CFAA? The Ninth Circuit's 2022 hiQ Labs v. LinkedIn decision is important because it held that the CFAA's "without authorization" concept does not apply in the same way to information on a public website that does not require authentication. That does not make scraping generally lawful: contractual and other legal issues remain separate. See the Ninth Circuit opinion.
Why does my scraper return status 999? LinkedIn can return HTTP 999 when it does not serve a request normally. Treat repeated 999 responses as an operational signal, not proof of one specific cause. Check request rate, network conditions, session behavior, and client configuration before changing one variable at a time.
Can I scrape LinkedIn without an account? Some public profiles, company pages, and job-search pages can be accessible without authentication, while other parts of the site require login. Availability can vary by page type and over time, so test the exact surface your application depends on.
Why did Proxycurl shut down? LinkedIn announced legal proceedings against Proxycurl in January 2025, citing unauthorized scraping and fake accounts. LinkedIn later announced on July 28, 2025 that the lawsuit had been resolved. The episode illustrates the legal and operational risk of commercial-scale scraping infrastructure. (LinkedIn, LinkedIn resolution announcement)
Is linkedin-scraper v3 backward compatible with v2? No. Version 3.0.0 replaced Selenium with async Playwright and switched data models to Pydantic. Pin pip install linkedin-scraper==2.11.2 to stay on v2 while migrating.
Which Chrome version should I impersonate? Use impersonate="chrome" when you want curl_cffi to track the latest supported Chrome impersonation target. The current curl_cffi documentation lists chrome146 as the latest available Chrome target and recommends the unversioned alias when you want the latest profile. Pin a version only when reproducibility matters. (curl_cffi)
Comments (0)
Login to post a comment.