RLHF & Alignment

Suppose you want a model to be helpful, honest and harmless. Try to write that down as a formula and you will fail — there is no clean equation for "a good answer." Yet you know one when you read it, and so does almost everyone else.

That is the whole idea behind RLHF — reinforcement learning from human feedback. Instead of defining the goal, we demonstrate it: show people two model answers and ask which they prefer. A small reward model learns to predict those judgments, turning a pile of human comparisons into a single number — a score for "how much a human would like this." The language model is then optimized to make that number go up.

RLHF is not a proof or a theorem; it is an engineering recipe, the one that took raw text-predictors and turned them into the chat assistants people actually use. It was popularized for deep learning by Christiano and colleagues (2017) and scaled to language models by OpenAI's InstructGPT (Ouyang et al., 2022).

Below are four candidate answers, each described by three simple traits: how helpful, honest and concise it is. Put them in your preferred order — best at the top — then press Train reward model. A tiny model adjusts three weights until the score it assigns matches the ranking you gave it.

<p class="hint">{{hint}}</p>
<ul id="cards" class="cards"></ul>
<div class="btns">
  <button id="train" type="button">{{btn_train}}</button>
  <button id="shuffle" type="button" class="ghost">{{btn_shuffle}}</button>
</div>
<div class="weights" id="weights"></div>
<div class="status" id="status">{{status_initial}}</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
.cards { list-style: none; margin: 0 0 .6rem; padding: 0; }
.card { display: flex; align-items: center; gap: .7rem; background: #eef2f6;
        border: 1px solid #cdd9e3; border-radius: 10px; padding: .55rem .7rem;
        margin: .35rem 0; cursor: grab; }
.card.drag { opacity: .4; }
.rank { font: 700 16px ui-monospace, monospace; color: #1d3557; width: 1.4em; text-align: center; }
.name { font-weight: 600; flex: 1; }
.traits { display: flex; gap: .35rem; }
.pill { font: 600 11px system-ui; background: #fff; border: 1px solid #cdd9e3;
        border-radius: 999px; padding: .1rem .45rem; color: #345; }
.score { font: 700 14px ui-monospace, monospace; color: #0a7d33; min-width: 3.2em; text-align: right; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin: .3rem 0 .6rem; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.weights { display: flex; gap: .8rem; flex-wrap: wrap; font-size: .85rem; color: #345; margin: .2rem 0; }
.weights b { color: #1d3557; }
.status { font-size: 1rem; font-weight: 600; margin: .5rem 0; min-height: 1.4em; }
.status.ok { color: #0a7d33; }
.status.bad { color: #c92f3c; }

// Code not found

Notice what the model never sees: a rule. You never told it "honesty matters twice as much as brevity." It infers the weights purely from your comparisons. Reorder the cards, retrain, and watch the learned preference shift. This is RLHF in miniature — the reward model is the learned stand-in for a human judge, and the assistant is later tuned to maximize it.

RLHF is powerful, but it is a heuristic with no correctness guarantee, and its weaknesses are well understood.

• The reward model is a proxy, not the truth. It is trained on a finite set of human comparisons, so it is only an approximation of what people actually value — and it is noisy, biased, and confidently wrong off-distribution.
• Optimizing a proxy too hard backfires. Push a policy to maximize an imperfect reward and it finds the cracks: outputs that score high but are not actually good. This is reward hacking, an instance of Goodhart's law — "when a measure becomes a target, it ceases to be a good measure."
• Human feedback is itself inconsistent. Annotators disagree, get tired, and prefer answers that sound confident over answers that are correct, so the signal we are fitting is fuzzy at its source.
• It is unsolved as a science. There is no theorem saying RLHF yields aligned behavior; making models robustly do what we mean — not merely what we measured — is an open research problem, closely tied to learning theory questions like PAC learning.

So RLHF's status is honest to state: empirically transformative, theoretically unguaranteed. It is the best recipe we have, not a solution to alignment.

"Learn the objective from human comparisons" turns out to be the missing piece for any task where the goal lives in people's heads:

• Chat assistants: the politeness, refusals and helpfulness you feel in a modern assistant are largely RLHF, layered on top of plain text prediction by neural-network training.
• Summarization and writing: "is this summary good?" has no formula, so preferences are gathered and a reward model trained.
• Code assistants: ranking which completion a developer accepts is a feedback signal in disguise.
• Robotics and control: the original 2017 work taught simulated robots backflips purely from humans clicking the better of two clips.
• Recommendation: choosing between items is the same comparison shape that drives recommendation systems.

Wherever you can't write the rule but can recognize the result, RLHF gives you a way to learn it.

RLHF is a beautiful sleight of hand: we cannot state our goal, so we demonstrate it through comparisons, fit a reward model to those judgments, and let the assistant climb that learned hill. It is the technique that made chatbots usable.

But the hill is a proxy, and proxies can be gamed. RLHF aligns a model with what we measured, which is only ever a shadow of what we meant. Closing that gap — making the learned reward faithful to human values even far from the training data — remains one of the open problems of AI, and a reason alignment is studied rather than declared finished.