This is a research-based piece, built from published papers and reporting — no lab of my own, no model trained. And the finding at its centre is almost too neat to believe: teach an AI only on what other AIs have written, and generation by generation, it starts to forget reality. Think of a photocopy of a photocopy of a photocopy. The first copy looks fine. By the tenth, the fine print has dissolved into grey smudges. In 2024, a team of researchers showed the same thing happens to AI models — and they gave it a name: model collapse.
The landmark study, published in Nature in July 2024 by Ilia Shumailov and colleagues, is disarmingly simple. The team fine-tuned a small language model on real text, then retrained each new generation only on the previous generation's output — no fresh human writing at all. With no original data retained, the model's perplexity (a measure of how confused it is by real text) rose by 20 to 28 points. Keep just 10% of the original human data in the mix, and the damage nearly vanished. The mechanism has three compounding sources: rare events simply never get sampled, the model has limited capacity, and the learning procedure itself is imperfect. Each generation loses a little of the edges, and the losses compound.
The paper's own example is the one to remember. It starts from a Wikipedia passage about English parish church towers. By generation five, the output has decayed into incoherent lists of languages. By generation nine, it is mostly one thing, repeated endlessly: "black-tailed jackrabbits, white-tailed jackrabbits, blue-tailed jackrabbits…" The model has not learned to lie. It has forgotten what variety looks like.
The first things to vanish in model collapse are the tails of the distribution — the rare, low-probability events. That sounds abstract until you think about what lives in the tails: minority languages, unusual medical conditions, niche skills, the long tail of human experience. The Nature paper makes the point directly: preserving the ability to model low-probability events is "essential to the fairness of their predictions", because such events are often relevant to marginalised groups. A model that has only ever read the average will only ever produce the average. The paper also warns that "the use of LLMs at scale to publish content on the Internet will pollute the collection of data to train their successors" — a feedback loop running in the open, right now.
Before you picture AI eating itself into oblivion, the follow-up research matters. A 2024 study by Gerstgrasser and colleagues confirmed collapse when each generation replaces the real data — but showed that accumulating synthetic data alongside the original real data avoids collapse across model sizes, architectures and modalities. A 2025 paper, "Collapse or Thrive?", confirmed the same contrast. But the debate is not settled: another 2024 paper, "Strong Model Collapse", found that in a supervised-regression setting, as little as 1% synthetic data in the training set could still trigger collapse, with bigger datasets not helping. And a 2025 position paper counted eight distinct, sometimes conflicting definitions of model collapse across 28 papers, arguing that many doomsday scenarios rest on unrealistic assumptions — nobody is actually deleting all the real data after each training run. Real in the lab, in other words; disputed as an apocalypse.
The reason this matters now is scale. A 2026 study using the Wayback Machine and an AI-text detector found that by mid-2025, roughly 35% of newly published websites were classified as AI-generated or AI-assisted, up from zero before ChatGPT's launch in November 2022 — about 13% fully AI-generated. The same study found no statistically significant evidence that this reduced factual accuracy or stylistic diversity, so far. And synthetic data is not inherently poison: DeepSeek's R1-Zero model, trained with reinforcement learning on verifiable rewards, lifted its score on a hard maths benchmark from 15.6% to 71.0%. The difference is verification. Synthetic data checked against ground truth is powerful; synthetic data replacing real data, unchecked, is the photocopy machine running unattended.
Here is the anecdote that made the whole problem visible to everyone outside AI research. In February 2024, the journal Frontiers in Cell and Developmental Biology published a paper on rat stem cells whose first figure — openly generated with Midjourney — showed a cartoon white rat gazing up at a dissected penis and testicles larger than the rest of its body, labelled with gibberish including "testtomcels", "Dissilced" and "dck". Readers flagged it on social media within hours, and the paper was retracted three days after publication, with the journal saying it did not meet its standards of editorial and scientific rigour. Frontiers' policy did allow AI-generated images if disclosed and fact-checked. The fact-checking plainly did not happen. Nobody was harmed, and the paper was not important — but the image became the internet's shorthand for what happens when AI output enters the record without a human checking it first.
Model collapse is not an AI apocalypse; it is a data-hygiene problem with known fixes — keep real data in the mix, and verify synthetic data against something true. The jackrabbits are a warning, not a prophecy. The internet is now full of machine-written text, and the models of the future will train on whatever we leave lying around. What we publish, and what we bother to check, is quietly becoming the training data of everything that comes next.
Related on Poptani Labs: How AI image generators actually make pictures — and why the hands go wrong, Why AI sometimes makes things up — and why it sounds so sure. More in this category: AI, Decoded.
Sources: Shumailov et al., 'AI models collapse when trained on recursively generated data', Nature 631 (2024), Gerstgrasser et al., 'Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data', Dohmatob et al., 'Strong Model Collapse', Doležal et al., 'The Impact of AI-Generated Text on the Internet', AI-generated rat figure retracted by Frontiers journal