In 2023 a New York lawyer filed a court motion that cited six legal precedents. The judge, reading through the brief, couldn't find any of them. Anywhere. They didn't exist. The lawyer had used ChatGPT, which confidently invented them — exact case numbers, quotes, the works. He got a $5,000 fine and a long professional conversation with his peers.
The really interesting part isn't that the AI was wrong. It's that the lawyer believed it.
Why this isn't really AI's problem
I've grown to dislike the phrase "AI lies". It's just not true. To lie you have to know what's true and consciously deviate from it. The language model doesn't do that.
The language model does exactly one thing: given the input text, it predicts the most likely continuation. If you ask "Who wrote War and Peace?", it doesn't look it up. It pattern-matches statistically what kind of text usually follows that kind of question. If it has seen the name Tolstoy enough in that context, that's what it will say. If it has seen little — or contradictory data — it will still say something. Because that's its job. It'll just say "Dostoevsky".
The exact same mechanism produces the right and the wrong answer. From the inside there is no difference. That's why the model can't signal its own uncertainty — not because it doesn't want to, it doesn't feel it.
The problem isn't hallucination, it's blind trust
This is the key. Hallucination can't be eliminated. It can be reduced, fenced in, validated, but never fully removed. The field has long known about it, measured it, researched it. The real problem starts when an organization wires AI into a process as if it didn't hallucinate.
The lawyer's mistake wasn't using ChatGPT. It was submitting it as a court filing without verifying the citations. If he had used the AI as a first draft and then checked the six citations in a legal database, none of this would have happened. 5 minutes of work. He didn't have time, because "it'll be fine".
And this isn't a lawyer problem. I see exactly the same thing in enterprise environments, weekly:
- The finance person sends the AI-generated monthly report on to the CEO without changes.
- The HR person posts the AI-generated job ad without compliance review.
- The support manager pushes the AI-suggested reply directly to the customer through the chatbot, without validation.
In all three cases the AI works. 90% of the time even well. The trouble is the other 10% — and the fact that there's no layer to catch it.
What is a 90% accuracy worth?
This question is worth more than the entire AI ROI debate. "The AI is right 90% of the time" — great. What do you do with the 10%?
For marketing copy: the human review catches it. 10% wrong, 5 minutes to fix, ship. You can enjoy the autonomy.
For an internal chatbot: the 10% is annoying but fixable. With RAG, source citation and an "uncertain → handoff to human" path it can come down to 1-2%.
For drug dosage suggestions: the 10% kills people. Here 90% isn't success. Here even 99.99% is up for debate. That's why there is no fully autonomous medical AI on the market.
So the real question is never how accurate is the AI. It's what is the cost of an error and which layer catches it. If the cost is low and you catch it fast, AI can run autonomously. If the cost is high, or only surfaces weeks later — what you build around it really matters.
Three things that are worth more than they sound
Year after year the same three things come up as the most useful defense against hallucination. None of them are sexy. None of them will win a conference talk. But together they can drag the error rate down to a perfectly acceptable level.
Give the model sources, don't let it remember. This is called RAG, and basically the entire "enterprise AI" industry is built on it. If you make the model answer questions from your own documents, and demand a source citation in the answer, most hallucinations never get a chance to form. The ones that do can be filtered with citation validation.
If it's uncertain, make it say so. Most prompts don't include the sentence "if you don't know for sure, say you don't know". Yet that's one sentence. With structured output you can go further: ask for a confidence field, and if it's low, don't even show the answer to the human — send it to a validator instead.
Don't let it do math. The model is excellent at writing and terrible at arithmetic. If you need math, dates, real-time data — give it a tool. Let it call a calculator, a database, a search API. Those will never hallucinate, because they don't invent anything, they execute.
The responsible adult
This is what I keep coming back to whenever someone says "but the AI got it wrong sometimes". Yes. It always will. The question isn't that, it's: who is the responsible adult next to it?
The model is a very creative child. It writes a poem, makes a summary, suggests a solution, drafts a strategy. But it doesn't know what's true and what's invented — and it will never learn that on its own. The adult is you. The architecture you build. The validation you don't skip. The human-in-the-loop you make mandatory at the critical points.
The 2023 lawyer wasn't a bad person. He didn't intend to lie to the court. He just forgot that the adult role was on him. There was no one else to look at the six citations. He was relying on himself and on the AI. And the AI smiled back at him uncertainly, as if it knew.
If you want to go deeper: we worked out the full mitigation toolkit — the 5 types of hallucination, RAG code examples, structured output schemas, logprobs-based confidence, LLM-as-a-judge patterns, the production checklist and a business risk matrix — in our knowledge base piece on AI hallucination mitigation.