Every blog has a half-life. You publish a post with ten links. Two years later, three of them are dead: a startup shut down, a docs site moved, a GitHub repo disappeared. Readers hit a 404 in the middle of your tutorial, and a page full of broken links looks abandoned. The cruel part is which posts rot first: your oldest ones, which are often the ones still bringing in traffic.
Link checkers exist, but they tend to have two problems. Hosted ones want to crawl your whole site. The ones you run yourself cry wolf: one site returns 403 to anything that isn't a browser, another is down for ten minutes, a third rate-limits you, and your report fills up with false alarms until you stop reading it. A report you ignore is worse than no report.
So let's build one that only bothers you about links that are actually broken. It's a single Python file, standard library only, and it follows five rules:
Check your own posts, not the web. It reads the files you already have instead of crawling a site.
Be polite. One request per second per host, a small worker pool, and it obeys
Retry-After.Don't cry wolf. A link has to fail on several separate runs before it's reported, and sites that block bots are listed separately instead of being counted as broken.
Remember things. Results live in SQLite, so healthy links are only rechecked about once a week.
Be useful. The report tells you which post contains the link, how long it has been failing, and links to an archived copy when one exists.
The big picture
flowchart TD
A["Extract outbound links from posts"] --> B["Dedupe URLs"]
B --> C{"Checked recently?"}
C -- yes --> H["Use cached result"]
C -- no --> D["Per-host rate limiter"]
D --> E["HEAD request"]
E -- "4xx or 5xx" --> F["GET, headers only"]
E -- ok --> G["Classify result"]
F --> G
G --> I[("SQLite cache")]
H --> J["Weekly report"]
I --> JTwo scheduled jobs drive it: a quiet daily run that does the checking, and a weekly run that prints the report. We'll get to the cron lines at the end.
Step 1: Find the links
The checker reads a folder of posts (.md, .mdx or .html) and pulls out every http(s) URL. If your posts live in a CMS or database, export them to Markdown first, or point the script at your built HTML folder.
# Links inside code fences, <pre>/<code> and inline code are examples, not references.
# (\x60 is a backtick; spelled this way so the script can sit inside a Markdown code fence.)
CODE = re.compile(r"\x60{3}.*?\x60{3}|~~~.*?~~~|<(pre|code)\b.*?</\1>|\x60[^\x60\n]*\x60", re.S | re.I)
URL = re.compile(r"""https?://[^\s<>"'\x60\]]+""")
def clean(url):
"""Drop #fragments and the punctuation or markdown ')' that sticks to the end."""
url = url.split("#")[0]
while url and (url[-1] in ".,;:!?*" or (url[-1] == ")" and url.count(")") > url.count("("))):
url = url[:-1]
return urlThree details matter more than they look:
Strip code first. A
curl http://localhost:3000inside a code fence is an example, not a reference. Without this step, a developer blog fills its report withlocalhostlinks.Clean the tail of each URL. A regex happily swallows the
)that closes a Markdown link, or the full stop at the end of a sentence. Theclean()function strips trailing punctuation, and strips a)only when it's unbalanced, so Wikipedia-style URLs like.../Foo_(bar)survive.Drop
#fragmentsand dedupe. The result is a dictionary ofurl -> {posts that use it}, so a link used in five posts is checked once and reported with all five.
Two skip lists keep the noise down. --internal yourblog.com skips links to your own domain, which your site's own tooling should cover. A plain-text ignore file handles everything else:
# hosts that block bots or sit behind a login: can't be verified automatically
linkedin.com
x.com
localhost
127.0.0.1
example.comA line is a hostname (subdomains match too), unless it contains a /, in which case it's treated as a URL prefix.
Step 2: Be polite
A checker that hammers one host is a bad citizen, and it gets you rate-limited, which creates exactly the false alarms we're trying to avoid. The fix is a tiny throttle that hands out time slots per host:
class Throttle:
"""Space out requests to the same host; different hosts never wait for each other."""
def __init__(self, delay):
self.delay, self.next_ok, self.lock = delay, {}, threading.Lock()
def wait(self, host):
with self.lock: # reserve the next free slot for this host
now = time.monotonic()
slot = max(now, self.next_ok.get(host, 0.0))
self.next_ok[host] = slot + self.delay
time.sleep(max(0.0, slot - now))
def back_off(self, host, seconds):
with self.lock:
self.next_ok[host] = max(self.next_ok.get(host, 0.0), time.monotonic() + seconds)Each caller reserves the next free slot for its host while holding a lock, then sleeps outside the lock. With a one-second delay, 30 links to the same site take about 30 seconds, while links to other hosts proceed in parallel. When a server answers 429 Too Many Requests, back_off() pushes that host's next slot out by its Retry-After value (capped at two minutes, 30 seconds if the header is missing or isn't a plain number).
Two more habits help. The work queue is interleaved by host (one link from each site, then the next round), so no worker gets stuck behind a long list for a single domain. And the User-Agent honestly says it's a link checker. Some sites block unknown agents, which is a trade-off: pretending to be Chrome gets fewer 403s, but being honest is the better default for a script that runs unattended.
Step 3: Check one URL
Two decisions make the checker accurate.
HEAD first, GET as a second opinion. A HEAD request returns headers without a body, so it's cheap. But some servers mishandle it and answer 405 or even 404. So any HEAD that comes back as an error is retried with GET, and we still never download the body (unless you turn on soft-404 detection, below). A network error gets one retry; a 429 waits and retries.
def fetch(url, cfg, throttle):
"""One hop. HEAD first, GET as a second opinion, one polite retry on 429 or a network error."""
host = urllib.parse.urlsplit(url).hostname
method = "GET" if cfg.soft404 else "HEAD" # soft-404 detection needs the page title
for attempt in range(3):
throttle.wait(host)
status, headers, body, err = request(url, method, cfg, 32768 if cfg.soft404 else 0)
if status == 429:
throttle.back_off(host, retry_after(headers))
elif status >= 400 and method == "HEAD":
method = "GET" # some servers mishandle HEAD
elif status == 0 and attempt == 0:
pass # network hiccup: try once more
else:
break
return status, headers, body, errFollow redirects by hand. Python's urllib follows redirects silently, which hides the one thing we want to know: was it permanent? A 301 or 308 means "update your link"; a 302 or 307 means leave it alone. So the opener refuses to follow redirects, and check() walks the chain itself:
def check(url, cfg, throttle):
"""Follow redirects by hand so we can tell a permanent move from a temporary one."""
current, permanent = url, False
for _ in range(cfg.max_redirects + 1):
status, headers, body, err = fetch(current, cfg, throttle)
location = headers.get("Location") if headers else None
if status in REDIRECTS and location:
permanent = permanent or status in (301, 308)
current = urllib.parse.urljoin(current, location)
continue
break
else:
return Result(0, current, "too many redirects", "flaky")
moved = permanent and not same_place(url, current)
suspect = cfg.soft404 and 200 <= status < 300 and looks_like_soft_404(body)
return Result(status, current, err, classify(status, err, moved, suspect))A few details in there are worth pointing out. A redirect chain is capped (five hops by default), which also stops infinite loops. And same_place() ignores the redirects that aren't worth an email: http to https, www versus bare domain, and a trailing slash.
def same_place(a, b):
"""http vs https, www vs bare domain and a trailing slash are not worth reporting."""
def key(u):
p = urllib.parse.urlsplit(u)
return (p.hostname or "").removeprefix("www."), p.path.rstrip("/"), p.query
return key(a) == key(b)Step 4: Decide what the result means
This is the part that stops it crying wolf. Every link ends up with one of six verdicts:
flowchart TD
R["Final HTTP response"] --> S2{"2xx?"}
S2 -- yes --> T{"Title says not found?"}
T -- yes --> SUSPECT["suspect"]
T -- no --> P{"Permanent redirect to a different page?"}
P -- yes --> MOVED["moved"]
P -- no --> OK["ok"]
S2 -- no --> D{"404, 410 or DNS failure?"}
D -- yes --> DEAD["dead"]
D -- no --> B{"401, 403, 451 or 999?"}
B -- yes --> BLOCKED["blocked"]
B -- no --> FLAKY["flaky"]def classify(status, error, moved, suspect):
if 200 <= status < 300:
return "suspect" if suspect else "moved" if moved else "ok"
if status in (404, 410) or (error or "").startswith("DNS"):
return "dead"
if status in (401, 403, 451, 999):
return "blocked" # the site refuses robots; can't tell
return "flaky" # timeouts, 429, 5xx, resets, redirect loopsVerdict | What it means | Counts as failing? | When you hear about it |
|---|---|---|---|
| 2xx response | No | Never |
| Permanent redirect to a different page | No | Listed under "Moved" so you can update the link |
| 200, but the page title says "not found" | No | Listed under "Worth a look" |
| 401, 403, 451 or 999: the site refuses bots | No | Listed as "could not verify" |
| 404, 410 or the domain no longer resolves | Yes | After 2 failing runs |
| Timeout, 429, 5xx, connection reset, redirect loop | Yes | After 3 failing runs |
The rule behind the last two rows is "three strikes, on separate days". A 404 is fairly convincing, but a deploy can briefly produce one, so it needs two failing runs. Timeouts and 5xx errors are far more often temporary, so they need three. Failing links are rechecked every 20 hours, so with a daily cron job a link has to fail on consecutive days before it reaches your inbox. A site that's down for an afternoon never does.
Step 5: Remember everything
All state lives in one SQLite table, and the checker only rechecks what's due:
def open_db(path):
db = sqlite3.connect(path)
db.execute("""CREATE TABLE IF NOT EXISTS links (
url TEXT PRIMARY KEY, status INTEGER, verdict TEXT, final_url TEXT, error TEXT,
last_checked REAL, first_failed REAL, fail_count INTEGER NOT NULL DEFAULT 0)""")
return db
def is_due(row, now, cfg):
if row is None:
return True
last_checked, fail_count = row
wait = cfg.retry_hours * 3600 if fail_count else cfg.ttl_days * 86400
return now - last_checked >= waitHealthy links are rechecked after 6 days (so a daily cron effectively rechecks them weekly), and failing links after 20 hours. The function that ties it together:
def run_checks(db, found, cfg):
now = time.time()
known = {u: (t, f) for u, t, f in db.execute("SELECT url, last_checked, fail_count FROM links")}
due = [u for u in found if is_due(known.get(u), now, cfg)]
log.info("%d links found, %d due for a check", len(found), len(due))
# Interleave hosts so no worker gets stuck behind 30 links to the same site.
by_host = defaultdict(list)
for u in due:
by_host[urllib.parse.urlsplit(u).hostname].append(u)
queue = [u for tier in zip_longest(*by_host.values()) for u in tier if u]
throttle = Throttle(cfg.delay)
with ThreadPoolExecutor(cfg.workers) as pool:
results = list(zip(queue, pool.map(lambda u: safe_check(u, cfg, throttle), queue)))
# If most checks never got an answer, *we* are probably offline. Don't poison the cache.
no_answer = sum(r.status == 0 for _, r in results)
if len(results) >= 10 and no_answer > len(results) / 2:
raise SystemExit(f"{no_answer} of {len(results)} checks got no response; "
"looks like a network problem. Nothing was recorded.")
with db: # one transaction: a crash keeps the old state
for url, r in results:
row = db.execute("SELECT fail_count, first_failed FROM links WHERE url = ?", (url,)).fetchone()
fails, since = row or (0, None)
if r.verdict in ("dead", "flaky"):
fails, since = fails + 1, since or now
else:
fails, since = 0, None
db.execute("""INSERT OR REPLACE INTO links
(url, status, verdict, final_url, error, last_checked, first_failed, fail_count)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)""",
(url, r.status, r.verdict, r.final_url, r.error, now, since, fails))Three things in run_checks() are doing quiet but important work:
Offline guard. If ten or more checks ran and more than half got no response at all, the script assumes it is offline, not the internet, and records nothing. Without this, one DNS hiccup on your server would mark every link as dead, and DNS failures count as dead.
One transaction. Results are collected in memory and written in a single
with db:block. If the script is killed halfway through, the cache keeps its previous state instead of a half-updated one.Safe workers. Each check runs through
safe_check(), which turns any unexpected exception into aflakyresult. One strange URL must never sink the whole run.
Optional: catch soft 404s
Some sites return 200 OK with a "Page not found" page. With --soft404, the checker switches to GET, reads the first 32 KB, and tests the <title> against a short regex:
SOFT_404 = re.compile(r"404|not found|no longer (available|exists)|can.?t be found|does(n.?t| not) exist", re.I)
def looks_like_soft_404(body):
m = re.search(rb"<title[^>]*>(.*?)</title>", body, re.I | re.S)
return bool(m and SOFT_404.search(m.group(1).decode("utf-8", "ignore")))This is a heuristic, and it can be wrong: a post titled "How to fix 404 errors in nginx" will match. So these links land in their own "Worth a look" section and never count as broken.
The report
Once you have a cache, the report is a read-only query. Here's what the weekly report looks like (example data):
# Link report - 2026-11-16
142 links in 31 posts: 2 broken, 1 moved.
## Broken (2)
- https://old-startup.example/docs/getting-started
HTTP 404, failing since 2026-11-03
used in: 2023/first-api.md, 2024/rate-limits.md
archived copy: http://web.archive.org/web/20231105000000/https://old-startup.example/docs/getting-started
- https://gone-project.example/
DNS: [Errno -2] Name or service not known, failing since 2026-11-09
used in: 2024/tools-i-use.md
## Moved - update these (1)
- https://docs.example.org/v1/intro
now at: https://docs.example.org/latest/intro
used in: 2025/setup-guide.mdFor every broken link, the script asks the Internet Archive's availability API for the closest saved copy:
def wayback(url):
"""Closest archived copy from the Internet Archive, or None."""
api = "https://archive.org/wayback/available?url=" + urllib.parse.quote(url, safe="")
try:
req = urllib.request.Request(api, headers={"User-Agent": "linkcheck/1.0"})
with urllib.request.urlopen(req, timeout=10) as resp:
snap = json.load(resp).get("archived_snapshots", {}).get("closest")
return snap["url"] if snap and snap.get("available") else None
except (OSError, http.client.HTTPException, ValueError, KeyError):
return NoneThe endpoint returns JSON with archived_snapshots.closest.url when a copy exists, and an empty object when it doesn't. Any network problem just returns None, so a failed lookup never breaks the report. You can skip it with --no-wayback.
One side benefit: because the weekly report is printed even when nothing is broken, an "all clear" email doubles as a heartbeat. If it stops arriving, the job itself is broken.
The full script
Save this as linkcheck.py. It needs Python 3.10 or newer and nothing else.
#!/usr/bin/env python3
"""linkcheck.py - find dead outbound links in your blog posts.
Standard library only. All state lives in one SQLite file.
python3 linkcheck.py ~/blog/posts --internal yourblog.com # check, then report
python3 linkcheck.py ~/blog/posts --internal yourblog.com --check-only # for cron: silent
python3 linkcheck.py ~/blog/posts --internal yourblog.com --report-only # print the report
"""
import argparse
import http.client
import json
import logging
import re
import socket
import sqlite3
import threading
import time
import urllib.error
import urllib.parse
import urllib.request
from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass
from datetime import date, datetime
from itertools import zip_longest
from pathlib import Path
log = logging.getLogger("linkcheck")
SUFFIXES = {".md", ".markdown", ".mdx", ".html", ".htm"}
REDIRECTS = {301, 302, 303, 307, 308}
DEAD_AFTER = 2 # consecutive failing runs before a 404/410/DNS failure is reported
FLAKY_AFTER = 3 # same, for timeouts, 5xx, 429 and connection errors
SOFT_404 = re.compile(r"404|not found|no longer (available|exists)|can.?t be found|does(n.?t| not) exist", re.I)
# ---------------------------------------------------------------- 1. extract links
# Links inside code fences, <pre>/<code> and inline code are examples, not references.
# (\x60 is a backtick; spelled this way so the script can sit inside a Markdown code fence.)
CODE = re.compile(r"\x60{3}.*?\x60{3}|~~~.*?~~~|<(pre|code)\b.*?</\1>|\x60[^\x60\n]*\x60", re.S | re.I)
URL = re.compile(r"""https?://[^\s<>"'\x60\]]+""")
def clean(url):
"""Drop #fragments and the punctuation or markdown ')' that sticks to the end."""
url = url.split("#")[0]
while url and (url[-1] in ".,;:!?*" or (url[-1] == ")" and url.count(")") > url.count("("))):
url = url[:-1]
return url
def load_ignore(path):
p = Path(path)
if not p.exists():
return []
lines = [ln.strip() for ln in p.read_text().splitlines()]
return [ln for ln in lines if ln and not ln.startswith("#")]
def ignored(url, host, rules):
"""A rule is either a hostname (matches its subdomains too) or a URL prefix."""
for rule in rules:
if "/" in rule:
if url.startswith(rule):
return True
elif host == rule or host.endswith("." + rule):
return True
return False
def extract(posts_dir, rules):
"""Return {url: {post, post, ...}} for every outbound link we should check."""
root = Path(posts_dir)
found = defaultdict(set)
for path in sorted(root.rglob("*")):
if path.suffix.lower() not in SUFFIXES:
continue
text = CODE.sub(" ", path.read_text(encoding="utf-8", errors="ignore"))
for m in URL.finditer(text):
url = clean(m.group())
try:
host = urllib.parse.urlsplit(url).hostname
except ValueError:
continue
if host and not ignored(url, host, rules):
found[url].add(str(path.relative_to(root)))
return found
# ---------------------------------------------------------------- 2. be polite
class Throttle:
"""Space out requests to the same host; different hosts never wait for each other."""
def __init__(self, delay):
self.delay, self.next_ok, self.lock = delay, {}, threading.Lock()
def wait(self, host):
with self.lock: # reserve the next free slot for this host
now = time.monotonic()
slot = max(now, self.next_ok.get(host, 0.0))
self.next_ok[host] = slot + self.delay
time.sleep(max(0.0, slot - now))
def back_off(self, host, seconds):
with self.lock:
self.next_ok[host] = max(self.next_ok.get(host, 0.0), time.monotonic() + seconds)
class NoRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, *args, **kwargs):
return None # surface 3xx as errors; we follow them by hand
OPENER = urllib.request.build_opener(NoRedirect)
def quote(url):
"""Escape spaces and non-ASCII characters, keep everything already escaped."""
return urllib.parse.quote(url, safe=":/?#[]@!$&'()*+,;=%-._~")
def request(url, method, cfg, body_bytes=0):
"""One HTTP request -> (status, headers, body, error). status 0 means 'no response'."""
req = urllib.request.Request(quote(url), method=method,
headers={"User-Agent": cfg.user_agent, "Accept": "*/*"})
try:
with OPENER.open(req, timeout=cfg.timeout) as resp:
body = b""
if body_bytes and "html" in resp.headers.get("Content-Type", ""):
body = resp.read(body_bytes)
return resp.status, resp.headers, body, None
except urllib.error.HTTPError as e:
e.close()
return e.code, e.headers, b"", None
except urllib.error.URLError as e:
kind = "DNS" if isinstance(e.reason, socket.gaierror) else type(e.reason).__name__
return 0, None, b"", f"{kind}: {e.reason}"
except (OSError, http.client.HTTPException, ValueError) as e:
return 0, None, b"", f"{type(e).__name__}: {e}"
def retry_after(headers):
try:
return min(float(headers.get("Retry-After", "")), 120.0)
except (AttributeError, TypeError, ValueError):
return 30.0
def fetch(url, cfg, throttle):
"""One hop. HEAD first, GET as a second opinion, one polite retry on 429 or a network error."""
host = urllib.parse.urlsplit(url).hostname
method = "GET" if cfg.soft404 else "HEAD" # soft-404 detection needs the page title
for attempt in range(3):
throttle.wait(host)
status, headers, body, err = request(url, method, cfg, 32768 if cfg.soft404 else 0)
if status == 429:
throttle.back_off(host, retry_after(headers))
elif status >= 400 and method == "HEAD":
method = "GET" # some servers mishandle HEAD
elif status == 0 and attempt == 0:
pass # network hiccup: try once more
else:
break
return status, headers, body, err
# ---------------------------------------------------------------- 3. decide what a result means
@dataclass
class Result:
status: int
final_url: str
error: str | None
verdict: str
def same_place(a, b):
"""http vs https, www vs bare domain and a trailing slash are not worth reporting."""
def key(u):
p = urllib.parse.urlsplit(u)
return (p.hostname or "").removeprefix("www."), p.path.rstrip("/"), p.query
return key(a) == key(b)
def looks_like_soft_404(body):
m = re.search(rb"<title[^>]*>(.*?)</title>", body, re.I | re.S)
return bool(m and SOFT_404.search(m.group(1).decode("utf-8", "ignore")))
def classify(status, error, moved, suspect):
if 200 <= status < 300:
return "suspect" if suspect else "moved" if moved else "ok"
if status in (404, 410) or (error or "").startswith("DNS"):
return "dead"
if status in (401, 403, 451, 999):
return "blocked" # the site refuses robots; can't tell
return "flaky" # timeouts, 429, 5xx, resets, redirect loops
def check(url, cfg, throttle):
"""Follow redirects by hand so we can tell a permanent move from a temporary one."""
current, permanent = url, False
for _ in range(cfg.max_redirects + 1):
status, headers, body, err = fetch(current, cfg, throttle)
location = headers.get("Location") if headers else None
if status in REDIRECTS and location:
permanent = permanent or status in (301, 308)
current = urllib.parse.urljoin(current, location)
continue
break
else:
return Result(0, current, "too many redirects", "flaky")
moved = permanent and not same_place(url, current)
suspect = cfg.soft404 and 200 <= status < 300 and looks_like_soft_404(body)
return Result(status, current, err, classify(status, err, moved, suspect))
def safe_check(url, cfg, throttle):
try:
return check(url, cfg, throttle)
except Exception as e: # one weird URL must not sink the whole run
return Result(0, url, f"{type(e).__name__}: {e}", "flaky")
# ---------------------------------------------------------------- 4. remember (SQLite)
def open_db(path):
db = sqlite3.connect(path)
db.execute("""CREATE TABLE IF NOT EXISTS links (
url TEXT PRIMARY KEY, status INTEGER, verdict TEXT, final_url TEXT, error TEXT,
last_checked REAL, first_failed REAL, fail_count INTEGER NOT NULL DEFAULT 0)""")
return db
def is_due(row, now, cfg):
if row is None:
return True
last_checked, fail_count = row
wait = cfg.retry_hours * 3600 if fail_count else cfg.ttl_days * 86400
return now - last_checked >= wait
def run_checks(db, found, cfg):
now = time.time()
known = {u: (t, f) for u, t, f in db.execute("SELECT url, last_checked, fail_count FROM links")}
due = [u for u in found if is_due(known.get(u), now, cfg)]
log.info("%d links found, %d due for a check", len(found), len(due))
# Interleave hosts so no worker gets stuck behind 30 links to the same site.
by_host = defaultdict(list)
for u in due:
by_host[urllib.parse.urlsplit(u).hostname].append(u)
queue = [u for tier in zip_longest(*by_host.values()) for u in tier if u]
throttle = Throttle(cfg.delay)
with ThreadPoolExecutor(cfg.workers) as pool:
results = list(zip(queue, pool.map(lambda u: safe_check(u, cfg, throttle), queue)))
# If most checks never got an answer, *we* are probably offline. Don't poison the cache.
no_answer = sum(r.status == 0 for _, r in results)
if len(results) >= 10 and no_answer > len(results) / 2:
raise SystemExit(f"{no_answer} of {len(results)} checks got no response; "
"looks like a network problem. Nothing was recorded.")
with db: # one transaction: a crash keeps the old state
for url, r in results:
row = db.execute("SELECT fail_count, first_failed FROM links WHERE url = ?", (url,)).fetchone()
fails, since = row or (0, None)
if r.verdict in ("dead", "flaky"):
fails, since = fails + 1, since or now
else:
fails, since = 0, None
db.execute("""INSERT OR REPLACE INTO links
(url, status, verdict, final_url, error, last_checked, first_failed, fail_count)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)""",
(url, r.status, r.verdict, r.final_url, r.error, now, since, fails))
# ---------------------------------------------------------------- 5. report
def wayback(url):
"""Closest archived copy from the Internet Archive, or None."""
api = "https://archive.org/wayback/available?url=" + urllib.parse.quote(url, safe="")
try:
req = urllib.request.Request(api, headers={"User-Agent": "linkcheck/1.0"})
with urllib.request.urlopen(req, timeout=10) as resp:
snap = json.load(resp).get("archived_snapshots", {}).get("closest")
return snap["url"] if snap and snap.get("available") else None
except (OSError, http.client.HTTPException, ValueError, KeyError):
return None
def section(verdict, fails):
if (verdict == "dead" and fails >= DEAD_AFTER) or (verdict == "flaky" and fails >= FLAKY_AFTER):
return "broken"
if verdict in ("dead", "flaky"):
return "watching" # failing, but not often enough to bother you
return verdict # ok, moved, suspect, blocked
def build_report(db, found, cfg):
rows = {r[0]: r for r in db.execute(
"SELECT url, status, verdict, final_url, error, first_failed, fail_count FROM links")}
groups = defaultdict(list)
for url in sorted(found):
if url in rows:
groups[section(rows[url][2], rows[url][6])].append(url)
else:
groups["unchecked"].append(url)
posts = {p for ps in found.values() for p in ps}
counts = ", ".join(f"{len(groups[k])} {label}" for k, label in [
("broken", "broken"), ("moved", "moved"), ("suspect", "worth a look"),
("blocked", "could not be verified"), ("watching", "being watched"),
("unchecked", "not checked yet")] if groups[k])
out = [f"# Link report - {date.today()}", "",
f"{len(found)} links in {len(posts)} posts: {counts or 'nothing to report'}.", ""]
def used_in(url):
return " used in: " + ", ".join(sorted(found[url]))
if groups["broken"]:
out += [f"## Broken ({len(groups['broken'])})", ""]
for url in groups["broken"]:
_, status, _, _, error, since, _ = rows[url]
why = f"HTTP {status}" if status else error
out += [f"- {url}", f" {why}, failing since {datetime.fromtimestamp(since).date()}",
used_in(url)]
copy = None if cfg.no_wayback else wayback(url)
if copy:
out.append(f" archived copy: {copy}")
out.append("")
if groups["moved"]:
out += [f"## Moved - update these ({len(groups['moved'])})", ""]
for url in groups["moved"]:
out += [f"- {url}", f" now at: {rows[url][3]}", used_in(url)]
out.append("")
if groups["suspect"]:
out += [f"## Worth a look - page says 'not found' but returns 200 ({len(groups['suspect'])})", ""]
out += [f"- {u}\n{used_in(u)}" for u in groups["suspect"]] + [""]
if groups["blocked"]:
out += [f"## Could not verify - the site blocks bots ({len(groups['blocked'])})", ""]
out += [f"- {u}" for u in groups["blocked"]] + [""]
return "\n".join(out)
# ---------------------------------------------------------------- main
def main():
ap = argparse.ArgumentParser(description="Find dead outbound links in your blog posts.")
ap.add_argument("posts", help="folder with your posts (.md, .mdx or .html files)")
ap.add_argument("--internal", action="append", default=[], metavar="HOST",
help="your own domain(s); links to them are skipped (repeatable)")
ap.add_argument("--db", default="linkcheck.sqlite")
ap.add_argument("--ignore", default="linkcheck-ignore.txt", help="hosts or URL prefixes to skip")
ap.add_argument("--check-only", action="store_true", help="check links, print nothing")
ap.add_argument("--report-only", action="store_true", help="print the report, check nothing")
ap.add_argument("--soft404", action="store_true", help="flag 200 pages whose title says 'not found'")
ap.add_argument("--no-wayback", action="store_true", help="skip archive.org lookups in the report")
ap.add_argument("--workers", type=int, default=8)
ap.add_argument("--delay", type=float, default=1.0, help="seconds between requests to one host")
ap.add_argument("--timeout", type=float, default=15.0)
ap.add_argument("--max-redirects", type=int, default=5)
ap.add_argument("--ttl-days", type=float, default=6, help="recheck healthy links after this long")
ap.add_argument("--retry-hours", type=float, default=20, help="recheck failing links after this long")
ap.add_argument("--user-agent", default="Mozilla/5.0 (compatible; linkcheck/1.0; personal blog link checker)")
ap.add_argument("-v", "--verbose", action="store_true")
cfg = ap.parse_args()
logging.basicConfig(level=logging.INFO if cfg.verbose else logging.WARNING, format="%(message)s")
found = extract(cfg.posts, load_ignore(cfg.ignore) + cfg.internal)
db = open_db(cfg.db)
if not cfg.report_only:
run_checks(db, found, cfg)
if not cfg.check_only:
print(build_report(db, found, cfg))
if __name__ == "__main__":
main()Running it
Try it by hand first:
python3 linkcheck.py ~/blog/posts --internal yourblog.com -vOn the first run every link is due, so it takes as long as your busiest host needs (30 links to one site at one second each is about half a minute). Read the report, move any bot-blocked hosts into linkcheck-ignore.txt, and fix the "moved" links, which are the easy wins.
Then schedule it. Two entries cover everything:
# check links every day at 06:00 (prints nothing unless something goes wrong)
0 6 * * * cd ~/linkcheck && python3 linkcheck.py ~/blog/posts --internal yourblog.com --check-only
# print the report every Monday at 08:00
0 8 * * 1 cd ~/linkcheck && python3 linkcheck.py ~/blog/posts --internal yourblog.com --report-onlyCron emails any output to MAILTO if your machine can send mail. Otherwise, pipe the report into whatever you already use: mail, a Slack webhook, a push notification. The --check-only job stays silent unless something goes wrong, such as the offline guard firing, and cron mails that too.
One trap: cron uses the server's timezone. 0 6 * * * means 06:00 UTC on a UTC box, which is 11:30 in India. I wrote about exactly this problem in an earlier post on cron jobs firing at the wrong time.
What it can't catch
A checker that's honest about its limits is easier to trust:
Content drift. The link still returns 200, but the domain was bought by a casino. No status code tells you that.
Paywalls, logins and JavaScript-only pages. The server says OK, but the reader sees something useless.
#anchors. Fragments are stripped, so it verifies the page exists, not that the section does.Bot-blocked sites. They end up under "could not verify". Check those by hand now and then, or add them to the ignore file.
What a bot sees. This almost always matches what readers see, but geo-blocked content can differ.
Where to take it next
Run it in CI. A scheduled GitHub Action against your posts repository can open an issue with the report instead of sending an email.
Add a
--fixmode that rewrites "moved" links in place. Do it carefully, with a dry-run first, because it edits your posts.Pair it with archiving. The availability API only reads. When you publish a post that cites something important, saving a copy through the Wayback Machine's separate "Save Page Now" feature gives you a fallback before the link ever rots.
Link rot is slow and silent, which is exactly why a small script that checks quietly in the background beats a one-off audit. Run it once, fix what it finds, and let cron do the remembering.
Comments(0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!