Why AI Makes Things Up (and How to Reduce Errors)

A client forwarded me a screenshot, annoyed: his AI assistant had just cited a law article that does not exist, number included. That is not a rare bug, it is how a poorly framed model normally behaves. An AI does not look for what is true, it produces what is plausible. Grasping that already solves half the problem. Here is what a hallucination really is, why it happens, and the three levers I use to bring errors down.

What a hallucination really is

The word is misleading. A hallucination is not the AI going haywire or dreaming. It is a wrong answer stated with the same confidence as a correct one: the model flags nothing, because from its point of view nothing has changed. It takes three shapes. Information invented outright (a figure, a date, a quote). Real information wrongly connected (the right regulation applied to the wrong case). And an answer that contradicts your own documents even though the information was right there. The problem is not that it gets things wrong, it is that it gets them wrong without saying so.

Why it happens

A language model does one thing: predict the most likely next word, over and over, from billions of texts it has read. At no point does it consult a database of facts or check what it claims. It computes what sounds right, not what is true. Most of the time plausible and true overlap, which is why it works so well. Hallucinations are the cases where the two diverge. There is an aggravating factor: many models are tuned to answer no matter what. Faced with a question it cannot answer, a poorly calibrated model will rather invent than admit it does not know.

An AI does not look for the truth. It looks for the most plausible next words. Usually that is the same thing. Not always.

The situations that trigger the most errors

• Questions outside its knowledge: yesterday's price, an internal detail, a recent event. The model does not have it, so it fills the gap.
• Precise numbers and identifiers: amounts, dates, reference numbers, law articles. This is where it invents most readily.
• Vague or poorly framed questions: the less clear the request, the more it improvises to fill the void.
• Niche topics barely present in its training: your sector, your trade jargon, your in-house procedures.
• Long conversations: as the exchange drags on, it loses track of the original context and ends up contradicting itself.

Lever 1: connect the AI to your sources

This is by far the most effective lever. Instead of asking the model to answer from memory, I first have it retrieve the relevant passages from your documents (contracts, procedures, product sheets), then ask it to answer only from those passages, with the source. That is the principle of RAG, generation grounded in your own data: the model no longer recites, it reads and cites. The result is a sourced answer in 2 seconds and 70% less searching for your teams, provided the documents are clean. If you are starting out, where to begin with enterprise RAG avoids the classic mistakes.

One warning all the same: RAG does not fix wrong or outdated documents. If your base holds three contradictory versions of the same procedure, the AI will faithfully cite the wrong one. Answer quality is capped by source quality, which is why I never skip the prerequisite: checking that your data is ready for AI.

Lever 2: the evals that catch drift

A system that works on demo day can drift silently three weeks later, after a model update or a change in your data. Without measurement, you will not see it happen. An eval is a set of questions whose correct answers you already know, rerun regularly. I build thirty to fifty real cases, with the expected answers and the known traps, and on every change I measure the share of correct answers, the share of inventions, and the share of well-placed "I don't know". That turns an impression ("it seems to work") into a number tracked over time. The full method is here: measuring an AI agent in production.

Lever 3: guardrails and human review

• A human in the loop on anything sensitive: sending a customer email, approving a refund, editing a record. The AI proposes, a human decides.
• Automatic guardrails: on certain topics (legal, medical, contractual commitments) the AI is not allowed to answer and hands off.
• The right to say "I don't know": a system that escalates when unsure beats one that invents to save face.
• A deliberately narrow scope: an assistant that does three things well beats one that claims to do everything and is wrong one time in five.

Tuned right, the whole thing gives a system that is autonomous on the simple cases and cautious on the rest: a support setup that runs 24/7 and escalates 50% fewer tickets to your teams, because it handles what it masters and passes on the rest. That is the core of what I call system reliability: not the absence of error, but errors that are rare, caught, and without serious consequence.

Zero hallucination: myth or target

Let's be blunt: zero hallucination does not exist, and I am wary of anyone who promises it. As long as a model generates text, it can generate false text. Selling the opposite is selling hot air. The right question is not "how do I remove every error" but "how do I bring the error rate below a threshold acceptable for this use, and catch the ones that remain".

Setting an acceptable reliability threshold

I always start from the stakes, not the tech, with two questions. What does an error that slips through cost? And how fast do we catch it? A wrong answer about opening hours is fixed with one message; a wrong answer about a contract clause can be expensive and only surface months later. So the threshold is set use by use: full autonomy when the stakes are low and the error easy to catch, systematic human review when the stakes are high and the error hard to correct, and in between, autonomy on the simple cases with escalation the moment the system leaves its zone of confidence. Once that threshold is set, everything becomes measurable: you know what you are aiming for, when you hit it, and when you drift. That is exactly the work I do in an audit: pinpoint where the AI can go wrong in your context, what it would cost, and the level of control each use calls for. Not to chase an impossible zero, but to aim for a risk that is known and held.