Rerun Until Green
A test fails in CI. You didn't touch that code, the failure looks unrelated, and you've watched this exact one flake before. So you hit rerun. It passes. You merge. Nothing about that sequence feels like a decision, which is the problem.
The test suite earns its keep on one claim: green means safe to ship. That's the whole trade. You accept the minutes of waiting and the occasional false alarm because a passing run is supposed to mean something specific, that the change didn't break anything the suite knows how to check. Every rerun-until-green spends a little of that claim. Not the test's correctness. The meaning of the color.
Here's the mechanism. A flaky test passes and fails on the same code. So when you rerun it and it goes green, you haven't learned the code is fine. You've learned the test landed on its passing branch this time. The green is real in the sense that the job exited zero. It's fake in the sense that made you install the suite in the first place. From the dashboard the two are identical.
What makes this erode instead of self-correct is that each rerun is locally reasonable. The failure really was probably the flake. You really don't have time to debug someone else's intermittent test to ship a one-line change. Reruns are cheap and a heisenbug is not, so the rational move in every individual case is to hit the button. Stack enough locally rational reruns and you've trained the whole team to read red as "try again" instead of "stop." The signal didn't degrade because anyone decided the suite didn't matter. It degraded because nobody did.
The tell is what happens to a genuine regression once the team is fluent in reruns. A real bug ships a test that fails every time, deterministically, which is exactly what the suite is for. But by then red has been reclassified. The reflex response to any red is the rerun, because that's been the correct response to the last forty reds. So the real failure gets one rerun, then another, then a "still flaky?" in the channel, and somewhere in there someone force-merges, because the working prior now says red means noise. The suite caught the bug on the first try. The team had already stopped listening.
The reason this persists is that the cost lands on the wrong ledger. A flaky test presents as a small tax on whoever hits it: a couple of wasted minutes, one rerun, move on. So it gets triaged like a small problem, low priority, someone will get to it eventually. But the minutes aren't the cost. The cost is that every green in the repo now means a little less, for everyone, until the flake is fixed or pulled. That cost is large and paid by the whole team, and it's invisible on the ledger that sets priorities, which only ever sees the two minutes in front of one person. A cost paid by everyone and billed to no one doesn't get fixed on schedule. It gets absorbed.
There's a quieter version that skips the button entirely. A test that's been flaky long enough stops being watched at all. It moves into the mental category of "that one," and its failures get filtered out before they reach conscious attention, the way you stop hearing a fan that's been running all day. At that point the test still runs, still costs compute, still shows up in the report, and tells you nothing, because the one reader it needed has learned not to look. That's worse than deleting it, because it looks like coverage.
The fix is boring and known: quarantine a flaky test out of the required set the moment it's identified, so red stays meaningful, and fix it on its own clock. What's worth noticing is why that rarely happens on time. Quarantine is a decision, and reruns let you avoid making one. The button is right there, it works every time, and it asks nothing of you. A test you can always get past is a test that has quietly stopped testing you.