How LLMs and coding tools work9 min read

Training Language Models to Follow Instructions with Human Feedback

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, et al. · 2022 · NeurIPS 2022 · Advances in Neural Information Processing Systems 35

The paper that turned a text predictor into an assistant: humans rank outputs, a reward model learns their taste, and the model is tuned against it.

Read the paper ↗

The short version

→A 1.3B-parameter model trained on human feedback produced outputs people preferred over a 175B GPT-3 — a hundred times fewer parameters.
→Three stages: supervised fine-tuning on human-written demonstrations, then a reward model trained on human rankings, then reinforcement learning against that reward model (PPO).
→It made models more truthful and less toxic on the measures tested, at a small cost on some standard NLP benchmarks — the 'alignment tax'.
→Every assistant you call is a descendant of this recipe. The model's preference for being helpful, its hedging, its refusals and its sycophancy all live in this stage, not in pretraining.

Subscription

The Library

Everything running a software business asks of you and none of it is code: onboarding that activates, churn you can actually see coming, marketing with no budget, SEO that finds buyers, and the AI-plus-Ahrefs workflow for writing the blog that feeds it. One new playbook a week, plus the whole archive from day one.

$9/ month · cancel any time

✓A new playbook every week — onboarding, churn, growth, SEO, content
✓The full back catalogue the moment you join, not drip-fed
✓Vendor explainers: Stripe, Supabase, Resend, Vercel, the AI APIs
✓The research shelf — real papers, decoded, on learning and on LLMs
✓One email when a new one lands, off by default in one click
✓Cancel any time; you keep reading to the end of the month

first — it takes ten seconds.

No card is asked for yet — the subscription is recorded and really opens your account. See all plans →