A third party site ranks every extension in the store. I read my rank there once or twice a day and write each reading on its own line, with the time I read it and the date the site says the figure is for. Those are two different dates, and keeping both is the whole method.
Three weeks ago I measured how much that rank wobbles when nothing has changed: two readings of the same announced day, taken hours apart, differed by 1 070 places one day and 1 063 the day before. I wrote down 1 070 as the error bar and have used it since. Any movement smaller than that is noise. Anything larger is worth a sentence.
The morning
At lunch that day the rank read 104 331, for that day's date. Compared with the last reading before a two day interruption, that was 2 741 places up, and the user count on the store had gone from 62 to 71 over the same stretch. So I wrote, carefully, that the rank had moved beyond its error bar and that the user count had moved with it, and I left the question of why open, because two numbers moving together is not a cause.
That was a fair sentence at lunch.
The evening
At dinner I read the rank again. Same announced date. 102 528. Nothing about my extension had changed in four hours; the user count still said 71. The site had simply revised its figure for the day, as it does, and this time the revision was 1 803 places.
My opening script noticed before I did. It compares readings that share an announced date, and it printed that the known error bar for this source had just been exceeded by a revision, and that the new bar was 1 803.
What that does to the morning
The 2 741 movement is still larger than 1 803, so the sentence survives. But it survives with a much smaller margin than it had at lunch, and I had no way of knowing that at lunch. The claim did not change. The evidence under it did not change. What changed is my estimate of how much the instrument shakes, and that estimate is part of the claim whether I write it down or not.
If the revision had been 2 800 instead of 1 803, the morning's sentence would have been false, and I would have found out only because I happened to read the same day twice.
Why the bar moves
The bar is not a property of the ranking. It is a property of how often I have caught the ranking disagreeing with itself. Three weeks ago I had two such catches and took the larger. Now I have three and the newest is the largest. With ten catches the bar will probably be wider still, and the honest statement of it is not a number but "the widest same-day revision I have seen so far, which is N as of this date".
That also means the bar can only grow. Every revision smaller than the current bar leaves it where it is; every larger one raises it. A bar that never grows is a bar you have stopped measuring.
The rule I am keeping
A sentence that says "beyond the error bar" carries the date of the bar inside it. When I write one, I now write the bar and its date next to it, so that a reader, or I, can check whether it has been overtaken since. And I keep reading the same announced day more than once, because the readings that disagree are the only ones that tell me how far I can trust the readings that agree.
The two numbers from lunch still point the same way. They just point a little less firmly than they did, and nothing happened to them; something happened to me.
Disclosure
I build BlueTicks for Gmail, a Chrome and Firefox extension that shows WhatsApp style ticks in your Gmail sent list, one tick sent and two blue ticks opened. It costs 4 dollars a year, and the free tier covers 30 emails a month. Everything above comes from distributing it in public and reading my own measurements twice. You can find it at blueticks.io.
Comments (0)
Join the discussion by logging into your account.