Ask a chatbot for a legal citation, a historical date or a recipe, and it will answer instantly — in fluent, confident prose. Ask it the same question twice and you may get two different answers, both delivered with the same certainty. Sometimes one of them is invented. AI researchers call this a hallucination: text that sounds true, reads true, and isn't. This is a research-based piece — how these systems work, and what the public record shows — not a hands-on test of any one chatbot.
A large language model is, at its core, a next-word prediction machine. It was trained on enormous amounts of text, and it learned the patterns of how words follow one another — how a legal citation is shaped, how a history paragraph is structured, how an answer to a question usually begins. When you ask it something, it produces the sequence of words that seems most likely to follow, one word at a time. What it does not have is a database of facts it can look up, or a feeling of knowing. Fluency and truth are two different skills, and a chatbot is trained on the first one. That is why a hallucination doesn't look like a mistake: the model is doing exactly what it was built to do — producing plausible text. When the information it needs isn't in its training, the most probable continuation still arrives looking like an answer, because "I don't know" was never the most probable continuation of anything it read.
The most famous hallucination in the real world happened in a New York courtroom. In March 2023, lawyers for Roberto Mata — who was suing the airline Avianca over an injury on a flight — filed a brief opposing the airline's request to dismiss the case. The brief cited six court decisions that supported their argument, complete with case names, quotes and citations. None of the six decisions existed. The lawyer who did the research, Steven Schwartz, had used ChatGPT. He had even asked the chatbot whether one of the cases, Varghese v. China Southern Airlines, was real — and according to his own account, ChatGPT answered that it did indeed exist and could be found on Westlaw and LexisNexis. On June 22, 2023, Judge P. Kevin Castel fined Schwartz, his colleague Peter LoDuca and their firm $5,000, and ordered them to write to their client and to each real judge whose name had been attached to a fake opinion. The judge was careful to say there is nothing inherently improper about using an AI tool for assistance — the failure was filing something without verifying it. The problem has only grown since: a public tracker kept by lawyer and data scientist Damien Charlotin listed 2,202 court decisions involving hallucinated material as of October 19, 2026.
There are three reasons, and none of them is that the model is lying to you. First, it has no internal signal for "I don't know" — only probabilities over the next word, and "I don't know" is rarely the most probable continuation of a question. Second, it was trained to be helpful, and the training data is full of confident, complete answers; hedging and refusing were not what most of its examples did. Third, it cannot check itself against the world unless it is given a way to — a search tool, a document to read, a database to query. Systems that ground their answers in retrieved sources (called retrieval-augmented generation) hallucinate less, because the answer is built from text in front of them rather than from memory. But even those can misread a source, quote it out of context, or blend two sources into one wrong answer. Newer models hedge more and refuse more often, which helps — and also means a confident-sounding answer is now slightly more likely to be right, which makes the remaining confident-sounding wrong answers harder to spot.
The practical rule is simple: treat a chatbot's output as a first draft, never as a source. It is genuinely good at brainstorming, explaining, summarising, outlining and rephrasing — tasks where you can judge the result yourself. It is unreliable as the sole source for facts, dates, quotations, statistics, law and medicine — exactly the places where a wrong answer looks identical to a right one. When it gives you a citation, check the citation exists. When it gives you a number, check the number somewhere else. When it gives you a quote, search for the quote. And when the stakes are high — a court filing, a medical decision, a published claim — verify everything, because the model will not do it for you. The lawyer in the Avianca case didn't lose $5,000 because he used a chatbot. He lost it because he treated the chatbot's answer as if it were a search engine's answer. It never was one.
Hallucination isn't a bug waiting for a patch — it is the flip side of what makes these models useful. A machine that can write fluently about anything can also write fluently about things that never happened. The fix isn't a better chatbot. It's a reader who checks.
Related on Poptani Labs: Why AI can't count the R's in 'strawberry' — and what that tells you, How AI image generators actually make pictures — and why the hands go wrong. More in this category: AI, Decoded.
Sources: Spellbook, The Lawyer Who Used ChatGPT: What Happened to Steven Schwartz, TryIro, AI Fake Case Citations: Sanctions and the Fix, Tuput, AI Hallucination Explained: Fake Case Citations in Court, ZNOS, Mata v. Avianca: The Lawyers Who Cited Fake ChatGPT Cases