The Checklist Was the Hard Part
Every week now there's a benchmark where a model "solves" something that sounds like research: a decades-old calculation, a formal proof of a hard theorem, an operating system built by a thousand agents working in parallel. Read past the headline and almost all of it is the same move. Someone handed the model a map. The model walked the map fast.
Finishing a problem someone else already mapped out and originating the problem's shape are different cognitive acts. The capability jump showing up in this latest round of reports is almost entirely in the first one. Worth saying plainly, because the headlines don't distinguish between them, and the distinction is the whole story.
Take a math formalization run where the model gets a checklist of intermediate theorems to hit before it starts. That checklist does something specific to the problem: it converts "find a path through an unknown space" into "confirm these N already-identified waypoints, in order." Those are not the same search. The second one is enormously easier, because most of the branching that makes a proof hard to find has already been pruned by whoever wrote the checklist. An order-of-magnitude speedup and a shorter proof on the second attempt are exactly what you'd expect from a smaller search space, not necessarily from a model that got better at discovery.
Same shape with the physics claims going around: a model "completing" calculations that were laid out years ago but "too laborious to grind through by hand." That phrasing is doing real work, and it's honest about what it concedes. Nobody is claiming the model noticed the open question, decided it mattered, or figured out what approach might crack it. Someone else did that part, years before any model existed to hand it to. What's left is the grind, and grinding fast is genuinely useful, but it's a different achievement than the one the headline implies.
Here's the cleaner contrast, from a human doing the opposite move on purpose. Someone building scoring software for a sport took on the part that was already mapped: the official rulebook, chapters already written down, formulas a spreadsheet was already fumbling through by hand. Turning that into code is finishing a known shape. Carefully, usefully, but known. Then there was a second, adjacent problem on the table: automated timing, photo finishes, the actual "who crossed first" call. He declined it, explicitly, in a parenthetical. That's the problem nobody had drawn the map for yet. Deciding what counts as simultaneous, what a sensor failure looks like, what you do when two cameras disagree, none of that had a checklist waiting. He knew the difference between the two kinds of hard and picked the one he could actually finish.
That's the test to run on every "AI solved X" claim going around right now. A thousand agents assembling an operating system from a known architecture, known driver interfaces, a known login screen, is finishing a design, not originating one, no matter how many agents or how few days it took. The actually hard version of that problem, deciding what an operating system needs to be when nobody has specified the machine yet, never shows up in that benchmark, because nobody is measuring it. Benchmarks measure problems someone already enumerated. That's what makes them benchmarks.
Which means the selection effect runs one direction, every time. A problem whose shape nobody has drawn doesn't get a leaderboard, because there's nothing yet to score against. So every capability claim built on a benchmark systematically undercounts the originating work and overcounts the finishing work, not from anyone's dishonesty, just from what's countable in the first place.
The checklist is still the expensive part. It's just expensive in a currency, sitting with an open problem long enough to know which three steps actually matter, that no leaderboard has a column for.