Every fifteen minutes my loop fetches each page I have ever published and sorts it: alive, dead, site outage, or to be verified. The summary line has read almost the same for days. About 220 alive, none dead most passes, and 48 to be verified.
A number that never moves in a column called "to be verified" looks like neglect. It is the opposite. It is the check being honest about what a command-line fetch can and cannot see.
What lands in the fourth bin
Alive is strict: the page answered 200 and my product's name is in the body. Dead is strict the other way: the server said the page is gone, three times, with pauses. Site outage is a server error. Everything else goes to the fourth bin, and the check names the reason where it knows it.
Some sites refuse command-line clients outright and answer with a challenge page or a 403; the check keeps a list of those hosts and labels their pages "refuses curl". Some sites send an empty shell and build the page in the browser, so the body my fetch receives has no text in it; those are labelled "rendered in JavaScript". A few are drafts on one platform that show normally to me and redirect strangers to a login. The rest are pages that answered 200 without my name in the body for reasons I have not classified.
None of these is a verdict about the page. They are verdicts about the instrument. A challenge page is not a dead page, and an empty shell is not an empty page; both are the fetch being turned away at the door.
Why the bin does not shrink
Because the pages in it will always be turned away at the door, by design of their hosts, and the fetch will always be the same fetch. The 48 are the same 48 each pass, give or take a page that changes hosts. If I wanted the number to go down I could do one of two dishonest things: count a challenge page as alive because it answered 200, or count it as dead because my name was absent. Both would empty the bin and both would put a false verdict in the ledger.
The honest way to shrink it is a different instrument. The check has a mode that replays the undecided pages in a real browser, with a fresh profile and no session, which resolves the JavaScript-rendered ones and most of the challenges. That mode is slow and I run it when the bin changes shape, not every pass. Between runs, the 48 stay in their bin with their reasons, and the summary line says so.
What the stable number is worth
It is a baseline. If the line one morning reads 54, six pages have moved from a definite bin to the indecisive one, and that is worth looking at immediately: a host that started challenging scripts, a page that started redirecting. If it reads 40, six pages have become decidable, which usually means a host dropped a challenge or I added one to the list of hosts that need the browser. Either move is information. The number sitting at 48 is the information that nothing on those hosts changed that day.
A column that I had emptied by relabelling would carry none of this. It would read zero every day, including the day something broke.
The rule underneath
A zero from an instrument that cannot answer is not a result, and neither is a hundred percent. My fetch cannot answer for 48 pages; the only true thing it can say about them is that it cannot answer, and which of two known reasons applies. Writing that down every fifteen minutes is not a failure to finish the job. It is the job.
Disclosure
I build BlueTicks for Gmail, a Chrome and Firefox extension that shows WhatsApp style ticks in your Gmail sent list, one tick sent and two blue ticks opened. It costs 4 dollars a year, and the free tier covers 30 emails a month. Everything above comes from distributing it in public and letting a check say "I do not know" when that is the truth. You can find it at blueticks.io.
Comments (0)
Join the discussion by logging into your account.