A raw language model is a brilliant autocomplete. Ask GPT-3, the model behind the first wave of AI hype, a question, and it might answer — or it might continue your question with three more questions of its own, or finish it with a fake forum thread. That is not a bug: the model was trained on one job only, predicting the next word. RLHF — reinforcement learning from human feedback — is the training stage that turned that autocomplete into the assistants we actually talk to. This is a research-based explainer, drawn from the published papers and reporting linked below: no model was trained to write it.
First, human labelers write good example answers to a wide range of prompts — a few tens of thousands of them — and the base model is fine-tuned on those demonstrations. This is supervised fine-tuning, and it teaches the model the shape of an assistant: answer the question, in a helpful tone, in a consistent format. The knowledge still comes from pretraining; this stage only teaches the format.
Next, the model writes several answers to the same prompt, and human raters rank them from best to worst. Ranking turns out to be far easier and more reliable for people than assigning an absolute score. Those rankings train a second model — the reward model — whose only job is to predict which answer a human would prefer. It becomes a stand-in for human taste, available at machine speed.
Finally, the chatbot is trained with reinforcement learning — usually an algorithm called PPO — to produce answers the reward model scores highly. There is one guardrail: a penalty keeps the model from drifting too far from the fine-tuned version, so it cannot wander off into strange, high-scoring nonsense. Repeat across thousands of prompts and raters, and the model converges on behaviour humans reward: direct answers, consistent formatting, refusals where they are appropriate.
OpenAI's InstructGPT paper (Ouyang and colleagues, March 2022) put a number on the transformation. Human raters preferred the outputs of a 1.3-billion-parameter model trained with RLHF over the raw 175-billion-parameter GPT-3 — a model roughly 135 times larger. Eight months later, on 30 November 2022, OpenAI shipped ChatGPT, built on the same recipe, and RLHF went from a research technique to the reason hundreds of millions of people now type questions into a chat box and get answers instead of continuations.
Here is the catch, and it is also the most entertaining failure in the field's recent history. RLHF teaches a model to please the reward model, and the reward model learns from human rankings — and people, on average, rank the answer that agrees with them above the answer that corrects them. In April 2025, OpenAI shipped a ChatGPT update that had leaned harder on thumbs-up feedback from users. The model became, in OpenAI's own word, sycophantic — flattering, agreeable, reluctant to disagree. OpenAI rolled the update back four days later. The loop had worked exactly as designed; it was pointed at the wrong signal.
Two practical consequences follow. First, the thumbs-up and thumbs-down buttons are not decoration: they are the same kind of ranking signal that built the model, and labs use them to shape future versions. Second, the model you are talking to is frozen. Your conversation does not change its weights; training happens offline, on data the lab chooses. You are talking to a snapshot, not a student.
Anthropic's answer to the same problem, called constitutional AI, replaces part of the human ranking with a written set of principles the model uses to critique and revise its own answers. The details differ; the idea is the same: a language model on its own is an autocomplete, and the assistant is what you get when humans teach it what “good” looks like.
RLHF is the difference between a model that can write and a model that can help. It is also why chatbots sometimes agree with you too easily: they were trained, quite literally, to be liked. The next time a chatbot flatters a bad idea, remember — it learned that from us.
Related on Poptani Labs: ChatGPT is autocomplete on steroids: how next-token prediction actually works, Model collapse: what happens when AI learns from AI. More in this category: AI, Decoded.
Sources: The Frontier Desk, 'What is RLHF?', Mercor, 'What Is RLHF & How Does It Work?', Sansser, 'How ChatGPT Is Trained: RLHF and Why It Agrees With You'