An AI hallucination is an answer that sounds certain and is false: an invented fact, a citation to a paper that does not exist, a function a library never had. Every chat model does it, including ChatGPT, Claude and Gemini. It is not a bug that will be patched away; it follows from how language models produce text. Prompts can make hallucinations rarer, and only checking can catch the ones that remain.
Why language models hallucinate
A language model writes one token at a time, each chosen because it is likely to follow the text so far. Likely, given everything the model saw in training. Nothing in that process looks a fact up. When the answer was common in the training data, the likely continuation is usually the true one. When it was rare, recent, or never there at all, the model still produces the most plausible continuation, and plausible text about a fact it does not have is a hallucination.
Some kinds of question are more exposed than others:
- Specific details: exact numbers, dates, names, version numbers, page references. The shape of the answer is easy to predict; the detail is not.
- Citations and links. A reference has a predictable format (authors, year, title, journal), so a model can produce a well-formed one for a paper that was never written.
- Rare or recent topics: a small library, a local regulation, anything after the model's training data ends.
- Questions with a false premise. Ask what a function does, and the model tends to answer as if the function exists.
Fluency is no signal of accuracy. A hallucinated answer has the same confident tone, the same formatting and the same level of detail as a correct one, which is why it is easy to miss.
The false-premise trap
The tabs below ask about DataFrame.smart_merge() in pandas. That method does not exist. The first question assumes it does; the second allows the model to say so.
DataFrame.smart_merge() merges two DataFrames and detects the join keys for you by comparing column names and data types. Use it when your frames share key columns with slightly different names, for example user_id and userId, since it can match them automatically.
For full control over keys and join type, merge() is still the better choice.
The first reply is a hallucination built from real parts: pandas does merge DataFrames, and key-matching is a reasonable thing to want. The question supplied the premise and the model supplied a plausible function to fit it. Current models catch false premises more often than older ones did, but a question phrased as an assumption still pushes the answer toward agreement.
The second prompt did two things. It asked whether the method exists before asking what it does, and it gave explicit permission to be unsure. The reply also handed back a way to verify, which is the habit to keep: hasattr checks the real library in one line.
Answer from the source, with quotes
The strongest way to reduce hallucination is to put the facts in the prompt. When the answer is in text the model can see, it does not need to recall anything, and you can check every claim against that text. Asking for a supporting quote for each claim makes the checking fast, and telling the model what to say when the document is silent makes it less likely to fill the gap with a guess.
How long production backups are kept: 30 days. The document gives 30 days for backups in general and 7 days only for staging. "Backups run every night at 02:00 UTC and are kept for 30 days."
Can I restore one myself: Not in the document. It only says how restores are requested: "Restores are requested in the #ops channel and are usually completed within four hours." That suggests another team performs them, but the document does not say whether self-service restore exists.
Look at how the reply treated the second question. A model answering from general knowledge might have described a restore command. Constrained to the document, it separated what the text says from what it only suggests. Switch off the constraints part and ask again to see whether the reply still keeps them apart.
The <document> tags keep the source material separate from your instructions; delimiters and XML tags explains why that matters, and context engineering covers feeding models the right documents at scale. The same idea powers chat apps with web search or file uploads: the answer is only as grounded as the sources in the conversation, so read the cited source, not only the summary of it.
Other prompt habits that help
- Ask, don't assume. "Is there a way to X?" before "How do I X?" A false premise is the easiest hallucination to trigger.
- Ask for uncertainty. "Mark any claim you are not confident about" gets you a list of places to check first. The labels are not calibrated probabilities, but they often point at the weak spots.
- Ask for sources you can open. Then open them. A citation you have not opened has not been checked, and a link can exist while saying something different from the claim it supports.
- Start fresh when a chat gets long. Details from early in a long chat can get mixed up or dropped as the conversation grows. For facts that matter, open a new conversation with the relevant material pasted in; tokens and the context window explains why.
- Ask a second time, fresh. If two independent answers to the same factual question disagree, at least one is wrong. Agreement is weaker evidence, because a model can repeat the same mistake.
Verify what matters
No prompt removes hallucination, so decide what needs checking by what it costs to be wrong. A brainstorm or a first draft can carry a few errors. A number in a report, a legal or medical claim, a quote attributed to a person, or a citation in your work needs a primary source.
Code is the easiest output to verify, because running it tests it. An invented method fails with an AttributeError or its equivalent as soon as the line that calls it runs, so make sure your test reaches that line. The risky case is an invented package name. If a model tells you to install a package you have never heard of, check that it exists, that it is the project you think it is, and that it is widely used, before you install it. Security researchers have shown that models often invent the same fake package names again and again, and an attacker who publishes a package under one of those names gets their code installed by anyone who follows the suggestion. The checks for AI-written code are covered in prompts for writing code.
Frequently Asked Questions
What is an AI hallucination?
An AI hallucination is an answer from a language model that sounds confident and fluent but is false or unsupported: a made-up fact, a citation to a paper that does not exist, a function a library never had. The model is not lying on purpose. It produced the text that looked most likely, and likely is not the same as true.
Why does ChatGPT make things up?
Chat models generate text by predicting the next token, one after another, based on patterns learned in training. Unless a tool such as web search or a document is in the conversation, nothing looks the answer up. When the right answer was rare in training data, or never there, the model still produces the most plausible continuation, and a plausible wrong answer reads exactly like a right one.
Can you stop AI from hallucinating completely?
No. Current models hallucinate less than earlier ones, and answering from documents you provide or from search results reduces it further, but no prompt removes it. Treat any fact, number, quote, citation or API that matters as a claim to check against a primary source.
Which prompts reduce AI hallucinations?
Three help most. Give explicit permission to say "I don't know". Provide the source material and ask the model to answer only from it. Ask for a direct quote supporting each claim, so you can check the claim against the text. Avoid questions that assume something is true, because the model tends to go along with the assumption.
What are examples of AI hallucinations in coding?
Calling a method a library does not have, passing an option that does not exist, importing a package nobody published, and describing behavior from an older version of a library as current. Code makes many of these easy to catch, because an invented method fails as soon as the line that calls it runs. Invented package names are the dangerous case: check that a package is real and well known before you install it.
