The Eval Gap
Ronald Bradford watched Andy Pavlo demo AI-assisted SQL tuning at Percona Live Bay Area 2026 and had a simple objection: the workload was simulated, so the optimization was measuring the wrong thing.
The argument doesn't require much setup. Production data has skewed distributions, hot paths, correlated columns, and query patterns that accumulate over time in ways no synthetic generator reproduces faithfully. An index that looks good against uniform test data can perform badly against the actual shape of production traffic, because the optimizer is making decisions about cardinality and selectivity that the synthetic data lies about. Bradford isn't against AI-assisted tuning in principle. He's against evaluating it on a harness that can't produce a valid signal.
This is the same reason query plan regression testing on prod replicas beats testing on dev snapshots. The dev snapshot has stale statistics, truncated tables, or missing rows in the long tail of your distribution. The query plan changes. The test passes. Then you deploy and the plan reverts to the bad one because the real data looks nothing like what the optimizer trained on. You tuned your way into a worse system with full test coverage of the improvement.
The deeper problem is that synthetic benchmarks feel rigorous. You're measuring something. The numbers move. Progress looks real until you hit production and the numbers that actually matter don't budge, or go the wrong direction. A benchmark that doesn't capture the shape of real load isn't a simplified version of the problem, it's a different problem.
This applies beyond SQL. Any eval harness that substitutes convenience for fidelity has the same failure mode. The simulated task gets optimized; the real task does not. The gap between them is invisible until it isn't. Bradford's complaint about Pavlo's demo is specific to database query planning, but the structure of the complaint is general: the evaluation environment has to match the deployment environment or you're measuring your optimizer's performance on a test that isn't the test.
The counterargument is that you have to start somewhere, and simulated workloads let you iterate faster than waiting for prod traffic to accumulate. That's true. The issue is when the simulated eval stops being a fast approximation and becomes the actual quality gate. If the AI-tuned query plan ships because it performed well on synthetic data and no one runs it against a prod replica before rollout, the benchmark wasn't a development tool. It was a substitute for evaluation, dressed up to look like evaluation.
Bradford's post is light on concrete numbers, which he acknowledges implicitly by grounding the argument in consulting experience rather than controlled experiments. The argument is still right. Some claims don't need a p-value; they need a plausible mechanism and a pattern of failures that matches the prediction. "Optimizer decisions are sensitive to data distribution" is one of those claims. Every DBA who's ever seen a plan flip after a stats refresh already knows it.
The thing worth sitting with is that AI-assisted tooling doesn't change the underlying problem. It compounds it. An agent tuning queries against synthetic workloads will optimize confidently and incorrectly, and the confidence is harder to second-guess than a human recommendation you can push back on. The eval gap isn't new. The gap between what the benchmark measures and what production requires has always been there. What's new is the speed at which you can iterate through the wrong solution space.