● A free public tool  ·  Built by Poptani Labs  ·  No sign-up  ·  Your data never leaves your browser
AI, Decoded · 5-minute read

Why AI sometimes makes things up — and why it sounds so sure

ChatGPT logo
Image: 'ChatGPT.png' by Lugab89, CC BY 3.0, via Wikimedia Commons

Ask a chatbot for a fact it doesn't know, and it will rarely say so. Instead you get a smooth, confident answer — sometimes complete with a citation or quotation that looks entirely real and isn't. This failure has a name borrowed from psychiatry: a hallucination. But the machine isn't seeing things — it's doing exactly what it was built to do.

It predicts words, not facts

A large language model is trained on enormous text to do one thing: predict the most statistically probable next word. Nothing in that objective checks whether a sentence is true. The model has no fact-checker inside, no database it consults, and no built-in sense of "I don't know".

That is why hallucinations read so well: the grammar is right, the tone is confident, the citation looks properly formatted. Researchers split the problem in two: hallucinations that contradict a given source, like a summary that quietly changes a date, and ones that can't be verified from any source at all.

Sources: Yale — Hallucinations, Why AI Hallucinates (Ji et al. 2023; Huang et al. 2023)

Why guessing beats saying "I don't know"

Here's the uncomfortable part: we trained the models to bluff. A 2025 paper bluntly titled Why Language Models Hallucinate, by researchers at OpenAI and Georgia Tech, argues that training and evaluation reward guessing over admitting uncertainty — most benchmarks give "I don't know" zero credit, so a confident guess scores better.

The same paper proves something stranger: even a perfectly trained model would still hallucinate on some facts. For "arbitrary" facts that appear only once in training data — a birthday, an obscure citation — a well-calibrated model must guess at roughly the rate those facts are rare. In one test, researchers asked a model for one author's birthday, telling it to answer only if it knew; across three attempts it gave three different wrong dates.

Why it matters: on rare, one-off facts, hallucination is partly built into how these models work — so manage it, don't expect it to vanish.

Source: Why AI Hallucinates — Kalai, Nachum, Vempala & Zhang (2025)

When it matters: the court cases that didn't exist

Courtrooms document it best, because judges write down what happened. In Mata v. Avianca in 2023, New York lawyers filed a brief citing court opinions that did not exist — fake quotations and citations, generated by ChatGPT. On 22 June 2023, the US District Court for the Southern District of New York fined the two lawyers and their firm $5,000, noting that using AI is not improper in itself: the lawyers, as signatories, had to verify what they filed.

Since then the cases have piled up: a public tracker listed more than 2,200 such court decisions by late October 2026. The pattern is always the same — the output looked real, nobody checked, and the signature, not the software, took the blame.

Source: AI Safety Association — documented hallucination cases

The anecdote that says it all

When the judge grew suspicious of the cited cases, one lawyer did something that should be taught in every classroom: he asked ChatGPT whether the cases were real. The chatbot confirmed its own fabrications. Asking the machine to check its own homework is not verification — it is a second confident guess wearing a lab coat.

The risky spots

Hallucinations cluster around specific requests. The US National Institute of Standards and Technology, in its Generative AI Profile (NIST AI 600-1, July 2024), calls the phenomenon "the production of confidently stated but erroneous or false content" — confabulation. The highest-risk targets are rare facts, exact numbers, dates, quotations, and citations: precisely the things that look most authoritative when delivered fluently.

It isn't limited to trivia. In a European Broadcasting Union and BBC study of 3,000+ news answers from four AI assistants, 45% had at least one significant issue and 20% had major accuracy problems. On OpenAI's PersonQA benchmark, reported hallucination rates ranged from 16% for one model to 48% for another. Confident delivery is not evidence of correctness.

Sources: NIST AI 600-1 and PersonQA figures, EBU/BBC news-answers study

How to catch it

The practical defences are unglamorous, and they work. First, verify citations independently — never trust a reference until you've opened it yourself, because fabricated citations are among the most common hallucinations. Second, cross-check anything load-bearing against a primary source: an official site, a published paper, a reputable newsroom. Third, treat confidence as styling, not evidence.

For people building with AI, the standard approach is grounding: connect the model to real documents (retrieval-augmented generation, or RAG), require every claim to cite a passage, and make "I don't know" a safe answer — "sources or abstain". RAG reduces hallucinations substantially but doesn't eliminate them: wrong retrieved context still yields wrong answers, and a model can contradict even the right context. That is why production systems add a verification layer that re-checks answers against sources. The same applies to everyday use: if it matters, check it somewhere that isn't the chatbot. This is also why AI agents that passed the demo can still fail in the real world — and why rogue-agent incidents keep coming back to bad grounding.

Sources: Layered hallucination guardrails, Grounding, citations, and abstention

What This Means

Think of a chatbot as a brilliant, fluent intern with no filing cabinet and a deep reluctance to admit it. Superb at explaining and drafting — genuinely dangerous as a sole source for facts or anything carrying your name. The skill that matters now isn't prompting; it's checking. Open the source. Confirm the date. Treat every confident answer as a claim to verify, not a fact to trust. Used that way, its fluency stops being a trap and becomes a fast first draft, with you as the editor who decides what is true.

More from AI, Decoded

← All posts