Introduction

Suppose you want a model to be helpful, honest and harmless. Try to write that down as a formula and you will fail — there is no clean equation for "a good answer." Yet you know one when you read it, and so does almost everyone else.

That is the whole idea behind RLHF — reinforcement learning from human feedback. Instead of defining the goal, we demonstrate it: show people two model answers and ask which they prefer. A small reward model learns to predict those judgments, turning a pile of human comparisons into a single number — a score for "how much a human would like this." The language model is then optimized to make that number go up.

RLHF is not a proof or a theorem; it is an engineering recipe, the one that took raw text-predictors and turned them into the chat assistants people actually use. It was popularized for deep learning by Christiano and colleagues (2017) and scaled to language models by OpenAI's InstructGPT (Ouyang et al., 2022).

Rank the Answers

Below are four candidate answers, each described by three simple traits: how helpful, honest and concise it is. Put them in your preferred order — best at the top — then press Train reward model. A tiny model adjusts three weights until the score it assigns matches the ranking you gave it.

<p class="hint">Each answer has three traits: <b>helpful</b>, <b>honest</b>, <b>concise</b>
(0&ndash;10). Drag the cards into <b>your</b> preferred order &mdash; best at the top &mdash;
then train. A reward model learns three weights so its score reproduces your ranking.</p>
<ul id="cards" class="cards"></ul>
<div class="btns">
  <button id="train" type="button">Train reward model</button>
  <button id="shuffle" type="button" class="ghost">Shuffle order</button>
</div>
<div class="weights" id="weights"></div>
<div class="status" id="status">Order the cards, then train.</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
.cards { list-style: none; margin: 0 0 .6rem; padding: 0; }
.card { display: flex; align-items: center; gap: .7rem; background: #eef2f6;
        border: 1px solid #cdd9e3; border-radius: 10px; padding: .55rem .7rem;
        margin: .35rem 0; cursor: grab; }
.card.drag { opacity: .4; }
.rank { font: 700 16px ui-monospace, monospace; color: #1d3557; width: 1.4em; text-align: center; }
.name { font-weight: 600; flex: 1; }
.traits { display: flex; gap: .35rem; }
.pill { font: 600 11px system-ui; background: #fff; border: 1px solid #cdd9e3;
        border-radius: 999px; padding: .1rem .45rem; color: #345; }
.score { font: 700 14px ui-monospace, monospace; color: #0a7d33; min-width: 3.2em; text-align: right; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin: .3rem 0 .6rem; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.weights { display: flex; gap: .8rem; flex-wrap: wrap; font-size: .85rem; color: #345; margin: .2rem 0; }
.weights b { color: #1d3557; }
.status { font-size: 1rem; font-weight: 600; margin: .5rem 0; min-height: 1.4em; }
.status.ok { color: #0a7d33; }
.status.bad { color: #c92f3c; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// A reward model in miniature. Each answer is a feature vector x = [helpful,
// honest, concise]. The model scores it as a weighted sum: r(x) = w . x.
// We never tell it the weights; it LEARNS them from the order the user gives.
// For every adjacent pair (better above worse) we want r(better) > r(worse).
// We minimise the Bradley-Terry / logistic pairwise loss with gradient descent:
//   L = sum over pairs  -log sigmoid( r(better) - r(worse) )
// Same shape as the preference loss used to train real RLHF reward models.
const ANSWERS = [
  { name: 'Detailed walkthrough', x: [9, 8, 3] },
  { name: 'Quick honest tip',     x: [6, 9, 9] },
  { name: 'Confident but vague',  x: [7, 3, 6] },
  { name: 'Short and correct',    x: [5, 8, 8] },
];
let order = [0, 1, 2, 3];        // indices, top = most preferred
let w = [0, 0, 0];

const sigmoid = z => 1 / (1 + Math.exp(-z));
const score = (x, ww) => x[0]*ww[0] + x[1]*ww[1] + x[2]*ww[2];

// Train weights so score(order[i]) > score(order[i+1]) for all adjacent pairs.
function train(epochs = 4000, lr = 0.02) {
  w = [0, 0, 0];
  for (let e = 0; e < epochs; e++) {
    const g = [0, 0, 0];
    for (let i = 0; i < order.length - 1; i++) {
      const a = ANSWERS[order[i]].x;   // preferred (above)
      const b = ANSWERS[order[i+1]].x; // less preferred (below)
      const diff = score(a, w) - score(b, w);
      const p = sigmoid(diff);         // model’s prob that a beats b
      const coef = (1 - p);            // dL/d(diff) = -(1 - p)
      for (let k = 0; k < 3; k++) g[k] += coef * (a[k] - b[k]);
    }
    for (let k = 0; k < 3; k++) w[k] += lr * g[k];
  }
}

// How many of the user’s adjacent preferences does the model now respect?
function agreement() {
  let ok = 0, tot = order.length - 1;
  for (let i = 0; i < order.length - 1; i++)
    if (score(ANSWERS[order[i]].x, w) > score(ANSWERS[order[i+1]].x, w)) ok++;
  return [ok, tot];
}

const TRAITS = ['helpful', 'honest', 'concise'];
function setStatus(t, cls) { const s = document.getElementById('status'); s.textContent = t; s.className = 'status ' + cls; }

function render() {
  const ul = document.getElementById('cards'); ul.innerHTML = '';
  order.forEach((idx, pos) => {
    const a = ANSWERS[idx];
    const li = document.createElement('li');
    li.className = 'card'; li.draggable = true; li.dataset.pos = pos;
    const pills = a.x.map((v, k) => `<span class="pill">${TRAITS[k]} ${v}</span>`).join('');
    const sc = (w[0]||w[1]||w[2]) ? score(a.x, w).toFixed(1) : '';
    li.innerHTML = `<span class="rank">${pos+1}</span><span class="name">${a.name}</span>`
                 + `<span class="traits">${pills}</span><span class="score">${sc}</span>`;
    ul.appendChild(li);
  });
  const wd = document.getElementById('weights');
  wd.innerHTML = TRAITS.map((t, k) => `<span>${t}: <b>${w[k].toFixed(2)}</b></span>`).join('');
  wireDrag();
}

function wireDrag() {
  let from = null;
  document.querySelectorAll('.card').forEach(card => {
    card.addEventListener('dragstart', e => { from = +card.dataset.pos; card.classList.add('drag'); });
    card.addEventListener('dragend', () => card.classList.remove('drag'));
    card.addEventListener('dragover', e => e.preventDefault());
    card.addEventListener('drop', e => {
      e.preventDefault();
      const to = +card.dataset.pos;
      if (from === null || from === to) return;
      const moved = order.splice(from, 1)[0];
      order.splice(to, 0, moved);
      render();
      setStatus('Order changed. Train to update the reward model.', '');
    });
  });
}

document.getElementById('train').addEventListener('click', () => {
  train();
  const [ok, tot] = agreement();
  render();
  if (ok === tot) setStatus(fmt('Reward model learned your ranking: it respects all %tot% preferences.', { tot: tot }), 'ok');
  else setStatus(fmt('Best fit found, but %v% of %tot% preferences can’t be met by a single linear reward.', { v: tot-ok, tot: tot }), 'bad');
  console.log('learned weights', w.map(v => v.toFixed(3)), 'agreement', ok + '/' + tot);
});
document.getElementById('shuffle').addEventListener('click', () => {
  for (let i = order.length - 1; i > 0; i--) { const j = (Math.random()*(i+1))|0; [order[i],order[j]]=[order[j],order[i]]; }
  w = [0,0,0]; render(); setStatus('Order shuffled. Train to fit a new reward model.', '');
});
render();
Result
Console

Notice what the model never sees: a rule. You never told it "honesty matters twice as much as brevity." It infers the weights purely from your comparisons. Reorder the cards, retrain, and watch the learned preference shift. This is RLHF in miniature — the reward model is the learned stand-in for a human judge, and the assistant is later tuned to maximize it.

The Real Difficulty

RLHF is powerful, but it is a heuristic with no correctness guarantee, and its weaknesses are well understood.

  • The reward model is a proxy, not the truth. It is trained on a finite set of human comparisons, so it is only an approximation of what people actually value — and it is noisy, biased, and confidently wrong off-distribution.
  • Optimizing a proxy too hard backfires. Push a policy to maximize an imperfect reward and it finds the cracks: outputs that score high but are not actually good. This is reward hacking, an instance of Goodhart's law — "when a measure becomes a target, it ceases to be a good measure."
  • Human feedback is itself inconsistent. Annotators disagree, get tired, and prefer answers that sound confident over answers that are correct, so the signal we are fitting is fuzzy at its source.
  • It is unsolved as a science. There is no theorem saying RLHF yields aligned behavior; making models robustly do what we mean — not merely what we measured — is an open research problem, closely tied to learning theory questions like PAC learning.

So RLHF's status is honest to state: empirically transformative, theoretically unguaranteed. It is the best recipe we have, not a solution to alignment.

Where It Matters

"Learn the objective from human comparisons" turns out to be the missing piece for any task where the goal lives in people's heads:

  • Chat assistants: the politeness, refusals and helpfulness you feel in a modern assistant are largely RLHF, layered on top of plain text prediction by neural-network training.
  • Summarization and writing: "is this summary good?" has no formula, so preferences are gathered and a reward model trained.
  • Code assistants: ranking which completion a developer accepts is a feedback signal in disguise.
  • Robotics and control: the original 2017 work taught simulated robots backflips purely from humans clicking the better of two clips.
  • Recommendation: choosing between items is the same comparison shape that drives recommendation systems.

Wherever you can't write the rule but can recognize the result, RLHF gives you a way to learn it.

Conclusion

RLHF is a beautiful sleight of hand: we cannot state our goal, so we demonstrate it through comparisons, fit a reward model to those judgments, and let the assistant climb that learned hill. It is the technique that made chatbots usable.

But the hill is a proxy, and proxies can be gamed. RLHF aligns a model with what we measured, which is only ever a shadow of what we meant. Closing that gap — making the learned reward faithful to human values even far from the training data — remains one of the open problems of AI, and a reason alignment is studied rather than declared finished.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/rlhf-alignment/Content licensed under CC BY-NC 4.0.