AI hallucinations: why confident answers can be wrong

Generating text does not automatically verify it

A language model generates tokens: small pieces of text that can be whole words, parts of words or punctuation. It uses patterns learned during training together with the current context to produce a continuation. Those patterns can encode useful factual relationships as well as the structure of convincing explanations.

But a fluent sentence is not a record of a fact-check. An answer can combine correct background information with an unsupported detail. NIST describes this risk as confabulation, including invented citations and misleading explanations that seem to justify a wrong answer.

[1][2]

A worked example: the invented return window

Imagine a fictional shop called Fern sells lamps. You ask a chatbot, ‘Can I return a Fern lamp after opening the box?’ It replies: ‘Yes, opened lamps can be returned within 30 days with a receipt.’

That answer contains several separate claims: opened items qualify, the deadline is 30 days, and a receipt is required. The wording is reassuring. None of those details is established by the question, and we have not supplied Fern's policy.

Now suppose the actual policy says opened lamps are eligible for 14 days, unless damaged. The chatbot's answer was plausible as a generic shop policy but wrong for Fern. Even if the receipt detail happened to be right, the deadline could still mislead someone.

This is a made-up example, not a measured result from a particular model. Its purpose is to show how to break a convincing answer into claims you can check.

Confident wording is not a confidence measurement

‘Definitely’ and ‘I am certain’ are parts of the generated response. By themselves, they tell you little about whether evidence supports the deadline. A detailed explanation can repeat the same unsupported assumption in several forms.

Return to the Fern answer and try removing the confident language. What remains is the claim ‘opened lamps can be returned within 30 days.’ That is the statement to verify. Length, tone and a tidy list of conditions do not settle it.

[2]

Why giving it the document helps, and where that can fail

Retrieval-augmented generation, or RAG, adds a search step: a system retrieves relevant material and supplies it to the model as context. The model can then use an external source when composing its answer. The original RAG research found more factual generated text than its comparison model on the tasks studied; that is evidence of improvement, not a guarantee for every answer.

For Fern, separate three questions: did the system find the current return policy, did it retrieve the paragraph about opened lamps, and did the answer preserve the 14-day limit and the exception? Finding a document solves only part of the problem. The wrong passage or an unsupported interpretation can still produce a wrong answer.

[3]

A common misconception: a citation proves the answer

A reference is a route to evidence. It can be invented, irrelevant or real but inconsistent with the claim. Open it, find the relevant passage, and compare what it says with the answer. A link labelled ‘Fern returns policy’ would not justify a 30-day deadline if the page says 14 days.

[2]

A small checking routine

For the fictional return-policy answer, the checking routine is straightforward:

• Separate the claims: eligibility, deadline, required proof and exceptions.
• Locate the shop's current policy rather than another shop's similar policy.
• Match each claim to a passage. Leave missing details unresolved instead of filling them in.
• If two passages conflict, identify that conflict before acting on the answer.