AI hallucinations: where they come from and how to reduce them

An AI hallucination — called confabulation in NIST documents — is content that sounds convincing but is false or unsupported by sources. It is a risk of the whole class of generative models, not of one provider alone; NIST links it to the mechanism of generating answers from the distribution of training data.

There is no single technique that guarantees the absence of such errors. A better question is: how can we reduce their frequency, detect a lack of evidence, and limit the consequences of a mistake?

Where hallucinations come from

There are four mechanisms, each requiring a different defence:

1. The model lacks the information it needs. You ask about your procedure, which the model has never seen. It fills the gap with what statistically fits.
2. The model received incomplete or contradictory information. There are two versions of a document, one out of date. The model will choose one and will not tell you that there were two.
3. The question assumes something untrue. “Which clause in our agreement regulates late-delivery penalties?” — when no such clause exists. The model will readily point to one because the question suggested that it exists.
4. The task requires precision the model does not inherently have. Calculating, quoting exact amounts, or citing from memory.

Adding relevant context mainly helps when information is missing, but it does not by itself solve contradictory sources, leading questions, or tasks that require deterministic precision. Those cases also need source control, carefully designed instructions, and safeguards outside the model.

Six techniques that genuinely work

1. Grounding in your own data

The basis is simple: before answering, the model receives relevant source excerpts. This is RAG. It can reduce errors caused by missing knowledge, but it does not guarantee correctness: retrieval may return the wrong passage and the model may interpret it incorrectly.

2. Requiring a source citation

An answer without a reference to the passage on which it relies can be rejected or routed to a human. A citation does not yet prove that the conclusion is right, but it enables a quick check and exposes the situation in which the system has no basis for an answer.

3. Explicit permission to say “I don’t know”

A model is optimised to generate an answer, so it needs an explicitly safe alternative: when the supplied passages contain no basis, the correct result is a statement that data is missing together with guidance on where to look. It is an inexpensive change to the instruction and flow logic, but its effectiveness needs testing with questions that have no answer.

4. Separating understanding from calculation

Arithmetic, comparisons, and hard rules are better performed in code, leaving the model to interpret the question and present the result. This reduces calculation errors, but you still need to validate the query generated by the model, data types, and the way the result is displayed.

5. A second verification pass

A separate step checks whether the answer actually follows from the supplied passages. It can be a rule, a comparison with structured data, or another model call with a narrowly defined task. A second model is not independent proof and can repeat the same error, so a high-stakes case needs a person or deterministic control. In multi-step work, a separate verifier role can be useful, but its value also needs measuring.

6. A human at the end where the stakes are high

The “system prepares, human approves” mode is not a failure of automation. It is a design decision about where to draw the line. For customer replies on contractual matters, the level of control should follow the risk, company policy, and legal assessment; the model itself should not be the final decision-maker.

One technique that only looks like a solution

An instruction in the prompt: “do not make things up; answer only on the basis of facts.”

It helps somewhat and is worth including, but treating it as a safeguard is a mistake. A prompt is guidance, not a technical constraint — the model may fail to follow it, especially when the question is leading or the context incomplete. The safeguard is that the system does not accept an answer without a source, not that we asked the model to be reliable.

How to measure whether it is better

Without measurement, all these techniques are just a story. A minimum measurement setup is:

1. A representative set of real questions from your process, with an expected answer, including questions for which the correct answer is “I don’t know.” A pilot can start with 30–50 cases if they cover the important categories and risks, then grow the set with detected mistakes.
2. After every change, count three things: correct answers, incorrect answers, and how often the system admitted it lacked data.
3. The metric to watch most carefully: how often the system answered confidently and incorrectly. This is a particularly risky error because it may go unnoticed. A false “I don’t know” also has a cost — it increases escalations and unresolved cases — so the two error types should be reported separately.

When hallucination risk disqualifies an idea

• When a mistake is irreversible. A transfer, an HR decision, a filing with an authority. Here we want code and a human.
• When nobody can judge whether the answer is correct. If no one can verify the result, you will not notice when the system begins to make mistakes.
• When source data is contradictory and nobody wants to organise it. The system will then present outdated information convincingly, which is worse than having no system.

Frequently asked questions

Does a stronger model hallucinate less?

It may, but the result depends on the task, language, data, and evaluation method. Do not assume an improvement from a model name alone — compare models on the same question set, and keep grounding and controls regardless of the result.

Can we reach zero hallucinations?

That should not be promised in an open generative system. A realistic design objective is measured risk, detection of unsupported answers, and restricted permissions so that one error cannot trigger an irreversible action.

Who should protect quality after implementation?

The process owner on your side, plus a recurring run of the evaluation set. An AI system’s quality can degrade without an obvious failure, for example after a change to the model, data, or user behaviour.

---

In a knowledge-based system, we design a rule that stops or escalates an answer without sufficient evidence, and we measure quality with a question set rather than an impression. See second brain for companies, or tell us which questions your system should handle.