Why do these things hallucinate?

№ 01

“Conversation” is a useful lie

• Looks like talking to a personThe interface is doing this on purpose — chat bubbles, typing indicators, polite tone. It’s not lying to you exactly, but it is using a frame that primes you to expect a person on the other end.
• Isn’tMagritte painted a pipe and wrote “this is not a pipe” under it. Because it wasn’t — it was a painting. Same move here. What’s on your screen looks like a conversation. It isn’t one.
• That gap → every failure mode todayThe distance between what we think is happening and what’s actually happening is where every failure mode in this talk comes from.

When you type into ChatGPT or Claude, it feels like a conversation. That feeling is the most important thing to unlearn in the next ten minutes.

Ceci n’est pas une conversation. This is not a conversation. Stick with me a second — there’s a Belgian painter from the ’20s, René Magritte, who pulled this exact gag. In 1929 he painted The Treachery of Images — a perfectly rendered pipe, with “this is not a pipe” written under it. Asked once whether it was actually a pipe, his answer was: “Could you stuff my pipe? No. It’s just a representation.” You can’t smoke a painting.

Same move here. What’s on your screen looks like a conversation. It isn’t one. The whole thing from here is about what it actually is, and why that gap — between what we think is happening and what’s actually happening — is where every failure we’re going to talk about comes from.

№ 02

What actually happens each turn

• No memoryThe whiteboard wipes clean between every message. Nothing persists in the model itself across turns — what looks like memory is just the transcript being re-read.
• Re-reads the whole transcript, every turnEvery time you hit enter, the entire conversation is fed back in as fresh input. The model treats turn 47 the same way it treated turn 1 — by reading everything from the top.
• Predicts one word at a timeSame machinery as your phone’s autocomplete bar — three plausible next words, one gets picked, repeat. Just running at much larger scale.
• Optimizes for plausible — not truePhilosopher Harry Frankfurt has a technical term for this output: bullshit. Not lying — lying requires caring what the truth is. The model produces what sounds right.

Here’s the mechanism. The model has no memory between your messages. Every time you hit enter, it re-reads the entire transcript from scratch, as if seeing it for the first time. Then it generates a reply one word at a time, each word chosen because it’s the most plausible next word given everything before it.

Notice the word — plausible, not true. There’s a philosopher named Harry Frankfurt who wrote a book about exactly this. In his technical sense of the word, he’d call this output bullshit — not because it’s wrong, but because the system has no relationship to truth in either direction. It’s not lying. Lying requires caring what the truth is. The model just produces what sounds right.

That distinction is going to do a lot of work in a minute.

№ 03

What it actually is

• A file of weightsHundreds of gigabytes of compressed regularities, sitting on a disk. Nothing’s happening inside it when nobody’s asking. It’s an object, not an agent.
• Doesn’t predict — gets sampledWe say “the model predicts.” Better: we sample the model. The verb belongs to whoever’s doing the sampling.
• Doesn’t know — gets queriedSame shape. We query the file; it surfaces what its weights happen to encode. “Knowing” isn’t on the menu.
• Doesn’t do anything — it just isWhen you’re not talking to it, the model isn’t sitting there waiting for you. It exists. That’s the whole verb.

So if it’s not a conversation, what is it? Honestly, a file. A big file of weights — distilled from the souls of children and using all the water you’ve ever seen.

The thing to notice is that the model doesn’t actually verb. We say “it predicts,” “it knows,” “it hallucinates” — those are all verbs, and the model isn’t doing any of them. It’s a file. We sample from it. We query it. We hand it inputs and read what comes out.

Every verb in the room belongs to us. The model exists. That’s about all it does.

№ 04

Where hallucinations come from

• Fluency over truthThe training objective rewards sounding right, not being right. Confidence and accuracy aren’t the same dial — and the model only has the confidence one.
• No “I don’t know” reflexImagine a Magic 8-Ball with all the hedging panels grayed out — only the confident answers remain. That’s the default.
• Stale knowledge after the cutoffThe weights are frozen at training. Anything that happened after — last week’s news, the latest version of any library, your changed mind — is invisible to it.
• Missing fact → invented substituteWhen the right answer isn’t in the model, it samples from the closest probable region of the distribution. The output sounds confident either way.

Now hallucinations make sense. They’re not glitches — they fall straight out of what we just described. The model is trained to sound right, not to be right. It has no built-in “I don’t know” reflex, so when it hits a gap, it fills the gap with something plausible — in the same confident tone as everything else. Anything past its training cutoff is invisible. And if a fact isn’t in the context window, it’ll quietly invent a substitute rather than flag the absence.

There’s actually a peer-reviewed philosophy paper from 2024 — out of the University of Glasgow — literally titled “ChatGPT is bullshit.” And the authors are using bullshit in the technical Frankfurt sense. Their argument is that “hallucination” is the wrong word entirely — because hallucination implies a failed perception, and the model isn’t perceiving anything. It’s just producing what sounds right.

The hallucination isn’t the model failing. It’s the model doing exactly what it was built to do.

№ 05

Where failures compound

• Anchors to your framing — even when wrongAsk a leading question, get a leading answer. The model treats your premise as part of the input it’s supposed to extend, not as a thing to question.
• Confidence ≠ accuracyTwo dashboard gauges: confidence pinned at 100%, accuracy bouncing around 60%. The needles don’t move together.
• Drift in long conversationsEarly instructions fade as later context dominates. By turn 30, your system prompt has less pull than what you said two messages ago.
• No verification unless we build itNothing in the system fact-checks the output. If verification is going to happen, you (or some tool you wire in) have to make it happen.

These are the subtler failures — the ones that don’t look like hallucinations but cost you the same. The model anchors to your framing, so if you ask a leading question, it’ll lean into your premise even when it shouldn’t. Its tone of confidence doesn’t track its actual accuracy — it sounds equally sure when it’s right and when it’s wrong. In long conversations, early instructions fade as later context dominates. And critically, nothing in the system checks the output against reality unless we explicitly build that in.

№ 06

The through-line

• Not bugs. Consequences.Hallucinations, drift, sycophancy — these aren’t TODOs the vendor will fix in the next release. They’re load-bearing properties of how the system works at all.
• “Conversation” → we expect memory, grounding, honestyThe word does the damage. As soon as we call it a conversation, we import assumptions the medium can’t keep.
• Model delivers fluencyFluency is what the model is genuinely great at. Treat that as the gift it is — and don’t ask it for the other things.
• We supply the rest — that’s the gapMemory, grounding, honesty — those parts come from us. The gap is real, but it’s where the work happens, not where the system is broken.

So here’s the through-line. Everything we just covered isn’t a list of bugs to be fixed. They’re consequences of the architecture. The word “conversation” sets us up to expect memory, grounding, and honesty — none of which the model actually has. What it delivers is fluency. We supply the rest of the meaning ourselves, often without realizing we’re doing it.

That is the gap.

The bridge looks like nothing until you step on it. The model’s output looks like truth until you check it.

№ 07

How to use it well

• Draft, not authorityFirst draft, every time. The output is a starting point for your judgment, not a substitute for it.
• Ground it (retrieval, tools, MCP)Feed the model real, verified data through retrieval, tools, MCP. Religious traditions have been doing exactly this for thousands of years — a fixed text loaded by a person who brings it to bear on the moment.
• Scope to fluency-shaped tasksGood fits: drafting, summarizing, reformatting, ideating. Bad fits: precise math, recent events, exact factual recall.
• Verify where being wrong costsAnywhere a confident-sounding wrong answer is expensive, build the check in. The model won’t do it for you.

Which lands us at the practical stuff. Treat the output as a draft, never as authority. Ground the model — feed it real, verified data through retrieval and tools, so it isn’t guessing.

Religious traditions have been doing a version of this for thousands of years. You’ve got a fixed text — call it the Bible, call it the Commandments — and then the people who load that text up have to bring it to bear on situations the original authors didn’t anticipate. That’s what a pastor does on Sunday morning. Takes the fixed text, brings it to this week, this congregation, this question. We didn’t invent that move; we rebuilt it. And honestly — the reason Claude and ChatGPT give you slightly different answers is the same reason a Baptist and a Methodist give you slightly different answers about the same verse. They came up in different traditions.

Scope the model’s work to fluency-shaped tasks — drafting, summarizing, reformatting, ideating. And anywhere being wrong has a real cost, build verification into the workflow.

But there’s a deeper principle underneath those four. Think about what you actually come away with from a useful session — not the output, the experience. A clearer articulation of the problem you didn’t know you had. A reframing of the question you brought. Permission to say the thing you were already going to say. The sense that the work has been engaged with.

That’s not the model providing that. That’s you, bringing your situation to something that gives you a surface to push against. It’s the same thing people come away with from prayer, from therapy, from a good talk with a friend. The vibe coding crowd has the right instinct, even if the name’s a little goofy — what you’re really doing is bringing a feel, a half-formed intent, into contact with a fluent surface and seeing what comes back. The output is the lesser half of that exchange.

You bring most of the value.

Which means the best users of these tools, like the best practitioners of any reflective practice, bring most of the value with them. The fix isn’t waiting for a smarter model. It’s becoming a better reader.

And conveniently — that’s what the rest of VibeFest is for. You’re about to spend it bringing your stuff to a fluent surface and seeing what comes back. This page told you why that works. The next few hours are where you go figure out how.

Welcome.