The Untyped Output Layer
The standard fix for hallucination is more scale, more training data, more RLHF passes teaching the model to say "I don't know." All of that treats the problem as a knowledge gap: the model didn't have the fact, so it guessed, so give it more facts. But a model that's seen the correct citation ten thousand times will still, on the wrong day, produce a fluent wrong one. That's not a coverage problem. It's a category error baked into how the output gets built.
Here's the mechanism. A transformer generates text one token at a time, and every token comes out of the same undifferentiated process regardless of what kind of claim it's part of. The token that completes "the capital of France is" and the token that completes "the DOI for that paper is" are produced by the identical operation: sample from a distribution shaped by pattern and context. Nothing in the architecture marks one span as "this must be verified against a source" and the other as "this can be pattern-completed." The model doesn't experience a difference between recalling and confabulating, because there is no difference at the level where the token gets chosen. Confidence, as expressed in output fluency, is not wired to correctness at all. It's wired to how well the local pattern fits.
This is why hallucinated citations look so specific. A real citation and a fake one draw on the same generative process, so a fabricated DOI has the same texture, the same plausible journal name, the same publication-year math, as a real one. If hallucination were a knowledge gap, you'd expect fake answers to look uncertain, hedgy, thin. Instead they often look better than the true ones, because the model is optimizing for "what does a citation look like" rather than "did this citation happen." Scale doesn't touch that gap, because scale improves the pattern library, not the mechanism that would let the model distinguish "answer from lookup" from "answer from pattern."
Compare this to a normal programming language, where a type system exists precisely to stop this kind of category confusion before runtime. You don't let a string masquerade as an integer because the compiler refuses to compile it; the boundary is enforced structurally, not by hoping the programmer remembers to check. Language models have no equivalent boundary anywhere in the generation path. Every span of output, whether it's a fact, an opinion, a made-up example, or a citation, is typed identically: token. There's no annotation carried forward from "this claim needs grounding" to "this claim doesn't," so nothing downstream can tell the two apart either.
This is also why retrieval and tool use actually help, and it's not for the reason people usually give. The story is usually "RAG adds facts the model didn't have." That's true sometimes, but it's not the load-bearing part. The load-bearing part is that retrieval forces a structural split in the pipeline: some span of the final answer is now required to originate from a document lookup rather than from the generation process, and the system can check whether that requirement was met. A citation with no matching retrieved passage can be flagged, dropped, or re-queried. That's a type check, crude and bolted-on, but a real one, applied at the only place anyone has bothered to build a seam. Tool use does the same thing for computation: a number that comes out of a calculator call is typed differently from a number the model would have guessed, and the calling code can enforce that difference. None of this makes the model smarter. It makes the pipeline capable of rejecting outputs that fail a constraint the model itself can't represent.
Once you see it this way, the recurring pattern of "just scale it" fixes not fixing hallucination stops being surprising. Scale improves the fluency and hit rate of the pattern-completion process. It does nothing to introduce a boundary that doesn't exist in the architecture. You could train a model to be right 999 times out of 1000 and it would still fabricate the thousandth time with exactly the same tone of voice, because rightness and tone were never coupled to begin with. Fixing that requires typing the output, not just improving the guesser, and so far the only people who've actually done it are the ones who forced part of the answer to come from somewhere outside the model.
That leaves an uncomfortable implication for anything that isn't wrapped in a retrieval or tool boundary. Free-form generation, unconstrained by any external check, has no internal mechanism stopping it from producing a hallucination with the exact same confidence as a fact. Not a worse mechanism. None.