How LLMs and coding tools work9 min read

Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, et al. · 2021 · arXiv preprint (OpenAI) · the Codex / HumanEval paper

The paper behind Copilot: a model fine-tuned on public code, and HumanEval — a benchmark that runs the generated code against tests instead of comparing it to a reference answer.

Read the paper ↗

The short version

→Codex is GPT-3 fine-tuned on publicly available code from GitHub; it is the model that shipped as GitHub Copilot.
→HumanEval is the paper's other, longer-lived contribution: hand-written programming problems scored by running unit tests, not by text similarity.
→On HumanEval the model solved 28.8% of problems in one attempt, versus 0% for GPT-3 and 11.4% for GPT-J — and far more when allowed many attempts.
→The gap between 'solves it in one shot' and 'solves it in a hundred shots' is the single most important number for anyone who reviews AI-written code.

Subscription

The Library

Everything running a software business asks of you and none of it is code: onboarding that activates, churn you can actually see coming, marketing with no budget, SEO that finds buyers, and the AI-plus-Ahrefs workflow for writing the blog that feeds it. One new playbook a week, plus the whole archive from day one.

$9/ month · cancel any time

✓A new playbook every week — onboarding, churn, growth, SEO, content
✓The full back catalogue the moment you join, not drip-fed
✓Vendor explainers: Stripe, Supabase, Resend, Vercel, the AI APIs
✓The research shelf — real papers, decoded, on learning and on LLMs
✓One email when a new one lands, off by default in one click
✓Cancel any time; you keep reading to the end of the month

first — it takes ten seconds.

No card is asked for yet — the subscription is recorded and really opens your account. See all plans →