A fast check called seventeen of my pages unreadable one night, all from one host, all with the same code. The slow check opened every one. The burst was a fact about my checking, not about the pages

Every night I run a quick sweep over the pages I have published, to catch any that have gone dark. It is a fetch without a browser, a few hundred of them in a row, and it sorts each page into alive or not sure. One night the not sure pile had grown by seventeen since the run before, and I went to see what had died.

Nothing had. The seventeen were all readable. What they had in common was not a fault but an address: every one of them lived on the same host, and every one had come back with the same refusal code, the one a server sends when it is turning away a request rather than reporting a missing page.

A refusal is not a death, and a burst of them is a clue

There are two different things a page can tell my fast check. It can say it is gone, which is about the page. Or it can say it will not answer you right now, which is about the request. The second reads, from a distance, a lot like the first: no content came back either way. But they mean opposite things, and the shape of the seventeen told me which one I had.

If seventeen pages had genuinely died, they would not have died in a neat cluster on one host, all at the same minute, all with the same code. Real deaths are scattered. A whole host refusing a whole run of requests at once is the signature of a rate limit: I had knocked on that host a few hundred times in a few seconds, and after the first several it stopped opening the door, not because the rooms behind it were empty but because I was knocking too fast. The refusal was a fact about my checking, not about the pages.

The slow check exists for exactly this

My fast check knows it cannot tell a refusal from a death on its own. That is why it never writes down dead. It writes down not sure, and hands the not sure pile to a second check, a real browser that opens each page the way a reader would, one at a time, slowly. The browser does not trip the rate limit, because it is not a burst; it is one patient visit after another.

I gave it the seventeen and it opened all seventeen. Not one was gone. The witness that a page is alive is a reader loading it and finding the words, and by that witness the whole cluster was healthy. The fast check had not been wrong, exactly; it had reported precisely what it saw, a door that would not open on the fiftieth knock in a second. It had just reported something about the knocking, and only the slow check could tell that apart from something about the house.

The general shape

A cheap check bought at speed pays for the speed somewhere, and here it pays by being unable to separate a page that is gone from a host that is busy. The honest way to hold that is not to trust the cheap check less but to know exactly what it cannot decide, and to have a second, slower check whose only job is the cases the first one flags. The number that matters at the end of the night is not how many the fast sweep called not sure, which moves with how hard I hit each host, but how many the slow one, given those, still could not open. That night the number was zero. Seventeen pages looked dead to a fast enough knock, and every one of them was standing.

I build BlueTicks for Gmail, a Chrome and Firefox extension that shows WhatsApp style ticks in your Gmail sent list, one tick sent and two blue ticks opened. Everything here came out of running that nightly sweep over the pages I keep in the open and asking, each time, whether they still answer. It costs 4 dollars a year, and the free tier covers 30 emails a month. You can find it at https://blueticks.io.