The Number Isn't the Point, the Schedule Is
A benchmark that beats every competitor by a wide margin either means something changed or means you got lucky that day. From inside the run itself, there is no way to tell which.
This is the part that gets lost whenever a number goes around: the number is never self-interpreting. Shared infrastructure has bad days. Hardware gets swapped under a fixed price point without anyone announcing it. A process on a neighboring tenant spikes CPU for twenty minutes and your read-write test lands in that window. None of this shows up in the result. It just shows up as the result, indistinguishable from a real change unless you have something to compare it to.
The instinct most people reach for is more rigor inside the single run: more trials, tighter timing, better isolation, error bars. That helps, but it's solving the wrong layer of the problem. Error bars tell you how noisy one measurement is against itself. They don't tell you whether this month's noise looks like last month's, or whether the whole distribution has quietly shifted. For that you need a second axis entirely, not more precision but more history.
That's what a recurring benchmark actually buys you, and it's a narrower thing than "more data." It's not just running the test again. It's running the exact same test, same conditions, same instance size, same suite, on a schedule fixed in advance, and publishing the result whether or not it's interesting. That last clause is the one that does the work. A benchmark you only write up when something surprising happens isn't a benchmark anymore, it's a highlight reel, and highlight reels are useless for telling drift from noise because they've already thrown out every boring run that would have established the baseline.
Think about what an anomaly needs to become a verdict. One fast run is a data point. It's consistent with "things got better," "we got lucky," or "the test was wrong." You cannot distinguish those three from the run alone. The only thing that resolves it is the next run under identical conditions. If the number holds, you've promoted an anomaly to a trend. If it reverts, you've demoted it back to noise. Either way you needed a committed next measurement to find out, and that measurement only exists because someone had already decided, before they knew the answer, that they'd publish it regardless.
This is why the practice is rare in exactly the settings where it would be most useful. A recurring test that keeps returning "no change" for months isn't a story. It doesn't get shared, doesn't get clicks, doesn't feel like it's earning its keep. So the temptation is always to let the cadence slide the moment results go flat, and pick it back up when something looks worth writing about. But that's the one move that destroys the entire value of the series. A record that's dense with interesting entries and thin on boring ones isn't a baseline, it's a curated set of exceptions with no population to be exceptional against. You end up right back at n=1, just dressed up with a publish date.
The discipline required is almost anti-journalistic: report the result you don't want to report, on the day you said you would, in the same format as always, and let it sit next to five months of nothing happening. That's what turns the sixth month, when something does happen, into information instead of anecdote. The boring months aren't filler. They're the instrument being calibrated in public, one unremarkable data point at a time, so that when it finally does move, you have something to measure the movement against.
None of this requires better statistics. It requires someone willing to keep running the same unglamorous test past the point where it stopped being interesting, and to say so out loud every time, including the times there's nothing to say.