Introduction

You hand a language model three examples — "cat → feline, dog → canine, eagle → ?" — and it completes the pattern. No training loop ran. No gradient updated a single weight. The model never even saw this mapping during pretraining. Yet it gets it right.

This is in-context learning (ICL): the ability of a large language model to adapt to a new task at inference time, guided only by examples embedded in the prompt. The weights are frozen; all the "learning" happens inside the context window.

ICL is strange enough to deserve a closer look. Classical machine learning separates training from inference sharply: you train once, then you predict. ICL blurs that line. The same forward pass that produces the answer is also, somehow, the act of picking up the pattern. Understanding why that is possible turns out to reveal something deep about what transformer attention is actually doing.

Try It

Each row below is one example the model sees before your query. Add or remove examples, change the rule, then hit Predict — the simulated model will complete the pattern using only what is in the context.

<!-- In-context learning demo: simulate few-shot pattern completion -->
<p class="hint">Add <b>input → output</b> examples below to teach the pattern, then enter a new input and hit <b>Predict</b>. No weights are updated — only the context changes.</p>
<div id="examples-container">
  <div class="section-label">Context (few-shot examples)</div>
  <div id="examples-list"></div>
  <button id="add-example" type="button" class="ghost">+ Add example</button>
</div>
<div class="query-row">
  <div class="section-label">Your query</div>
  <div class="query-line">
    <input id="query-input" type="text" placeholder="new input…" autocomplete="off" />
    <span class="arrow">→</span>
    <span id="prediction" class="prediction-box">—</span>
  </div>
</div>
<div class="btns">
  <button id="predict-btn" type="button">Predict</button>
  <button id="reset-btn" type="button" class="ghost">Reset</button>
</div>
<div id="status" class="status"></div>

/* Layout and color styles for the ICL demo. */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .8rem; line-height: 1.5; }
.section-label { font-size: .75rem; font-weight: 700; text-transform: uppercase;
                 letter-spacing: .05em; color: #5a7088; margin-bottom: .35rem; }
#examples-container { background: #f4f7fa; border-radius: 10px; padding: .7rem .85rem .5rem;
                      margin-bottom: .7rem; }
#examples-list { display: flex; flex-direction: column; gap: .35rem; margin-bottom: .45rem; }
.example-row { display: flex; align-items: center; gap: .4rem; }
.example-row input { flex: 1; padding: .32rem .55rem; border: 1px solid #cdd9e3; border-radius: 6px;
                     font: 14px ui-monospace, monospace; background: #fff; }
.example-row .arrow { color: #5a7088; font-size: 1rem; }
.example-row .del-btn { background: none; border: none; color: #c92f3c; font-size: 1.1rem;
                        cursor: pointer; padding: 0 .2rem; line-height: 1; }
.query-row { margin-bottom: .7rem; }
.query-line { display: flex; align-items: center; gap: .5rem; }
.query-line input { flex: 1; padding: .38rem .6rem; border: 1.5px solid #1d3557; border-radius: 6px;
                    font: 14px ui-monospace, monospace; background: #fff; }
.arrow { color: #5a7088; font-size: 1rem; }
.prediction-box { flex: 1; min-height: 2rem; display: flex; align-items: center;
                  padding: .32rem .6rem; background: #e8eef3; border-radius: 6px;
                  font: 600 14px ui-monospace, monospace; color: #1d3557; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin-bottom: .4rem; }
button { font: 600 14px system-ui, sans-serif; padding: .42rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.status { font-size: .9rem; font-weight: 600; min-height: 1.3em; color: #0a7d33; }
.status.bad { color: #c92f3c; }

// In-context learning: adapt to a new task from prompt examples alone.
// The real mechanism uses induction heads in transformer attention layers.

const DEFAULT_EXAMPLES = [
  ['cat', 'feline'],
  ['dog', 'canine'],
  ['eagle', 'avian'],
];

let examples = DEFAULT_EXAMPLES.map(e => [...e]);

// Render the editable example rows.
function renderExamples() {
  const list = document.getElementById('examples-list');
  list.innerHTML = '';
  examples.forEach((pair, i) => {
    const row = document.createElement('div');
    row.className = 'example-row';
    const inInput = document.createElement('input');
    inInput.type = 'text';
    inInput.value = pair[0];
    inInput.placeholder = 'input';
    inInput.addEventListener('input', () => { examples[i][0] = inInput.value; });
    const arrow = document.createElement('span');
    arrow.className = 'arrow';
    arrow.textContent = '→';
    const outInput = document.createElement('input');
    outInput.type = 'text';
    outInput.value = pair[1];
    outInput.placeholder = 'output';
    outInput.addEventListener('input', () => { examples[i][1] = outInput.value; });
    const del = document.createElement('button');
    del.className = 'del-btn';
    del.textContent = '✕';
    del.title = 'Remove example';
    del.addEventListener('click', () => { examples.splice(i, 1); renderExamples(); });
    row.appendChild(inInput);
    row.appendChild(arrow);
    row.appendChild(outInput);
    row.appendChild(del);
    list.appendChild(row);
  });
}

document.getElementById('add-example').addEventListener('click', () => {
  examples.push(['', '']);
  renderExamples();
});

// Predict: exact match first, then majority-vote fallback.
// This mirrors induction heads: copy the output that follows the best-matching input.
function predict(query) {
  const q = query.trim().toLowerCase();
  if (!q) return '';
  // exact match — induction head finds the last identical token in context
  for (let i = examples.length - 1; i >= 0; i--) {
    if (examples[i][0].trim().toLowerCase() === q) return examples[i][1].trim();
  }
  // fallback: vote on outputs seen in examples
  const freq = {};
  examples.forEach(([inp, out]) => {
    if (!inp || !out) return;
    const o = out.trim();
    freq[o] = (freq[o] || 0) + 1;
  });
  if (Object.keys(freq).length === 0) return '';
  // return the most common output (majority vote)
  return Object.keys(freq).reduce((a, b) => freq[a] >= freq[b] ? a : b);
}

document.getElementById('predict-btn').addEventListener('click', () => {
  const q = document.getElementById('query-input').value;
  const result = predict(q);
  const predBox = document.getElementById('prediction');
  const status = document.getElementById('status');
  if (!q.trim()) {
    predBox.textContent = '—';
    status.textContent = 'Enter a query first.';
    status.className = 'status bad';
    console.log('Predict called with empty query.');
    return;
  }
  if (!result) {
    predBox.textContent = '—';
    status.textContent = 'Add at least one example with an output.';
    status.className = 'status bad';
    console.log('No usable examples found.');
    return;
  }
  predBox.textContent = result;
  status.textContent = 'Predicted from context — no weights changed.';
  status.className = 'status';
  console.log('ICL prediction:', q, '→', result);
});

document.getElementById('reset-btn').addEventListener('click', () => {
  examples = DEFAULT_EXAMPLES.map(e => [...e]);
  renderExamples();
  document.getElementById('query-input').value = '';
  document.getElementById('prediction').textContent = '—';
  document.getElementById('status').textContent = '';
  document.getElementById('status').className = 'status';
  console.log('Demo reset to defaults.');
});

renderExamples();
Result
Console

Notice that changing even one example rewires the answer immediately. The "model" here uses simple frequency counting — a toy version of what real induction heads do when they copy patterns from earlier in the context. What is striking in both cases is that no parameter was touched: the behavior changed purely because the context changed.

The Real Complexity

How does in-context learning work mechanistically? The leading explanation centers on induction heads — a specific two-layer attention circuit described by Olsson et al. (2022).

  • What an induction head does. Layer 1 builds a map of which token follows which. Layer 2 looks up the current token in that map and copies whatever came after its most recent occurrence. In sequence A B … A → ?, the head predicts B.
  • Why that's enough for ICL. Given few-shot examples input₁ → output₁, input₂ → output₂, … inputₙ → ?, the induction circuit matches inputₙ to the earlier occurrences of similar inputs and copies their following output tokens.
  • The gradient-descent connection. Akyürek et al. (2022) showed that for linear tasks, a transformer running ICL implicitly implements a form of gradient descent in activation space — the forward pass is a learning step.
  • It is not fine-tuning. ICL produces no persistent change: start a new conversation without the examples and the "learning" vanishes. The model has not been updated; it has been steered.
  • Limits. ICL breaks when the required mapping conflicts strongly with pretraining priors, when the context window is too short, or when the task demands multi-step reasoning that exceeds what a fixed-depth forward pass can implicitly compute.

The phenomenon is well-understood at the mechanistic level (Olsson et al., 2022; Akyürek et al., 2022), though questions about its precise sample-efficiency bounds and failure modes remain active research.

Where It Matters

In-context learning is the engine behind almost everything interesting that large language models do at deployment time:

  • Prompt engineering: every few-shot prompt is an ICL invocation. Curating better examples can flip accuracy from 50% to 90% without touching a model parameter.
  • Chain-of-thought reasoning: showing the model how to reason through solved examples before posing a hard problem is ICL applied to reasoning steps, not just answers.
  • Classification without a head: add labeled examples to the context and the model classifies new inputs without any fine-tuned softmax layer.
  • Tool-use and agents: giving an agent examples of how to invoke a tool in the prompt is ICL applied to action sequences — the same pattern-copying mechanism governs when to call a function.
  • Rapid prototyping: teams ship new behaviors in hours by writing examples, not in weeks by curating datasets and running training jobs.

ICL also reveals the limits of prompt-only adaptation. It cannot inject new factual knowledge (the weights have not changed), and its "learning" evaporates when the session ends. For durable or high-accuracy tasks, transformer fine-tuning or retrieval augmentation is still the right tool.

Conclusion

In-context learning is one of the most surprising properties to emerge from scaling language models. The same network that was trained on next-token prediction turns out to harbour a gradient-descent-like learning algorithm inside its attention heads — one that activates whenever it sees examples in the context, and silently copies patterns that fit.

The mechanism is real, the circuit is identified, and the applications are everywhere. But remember the boundary: ICL steers a frozen model; it does not change it. The moment the prompt window closes, the "learning" disappears. For everything that needs to persist, you still need the optimizer — and for the deep theory of why transformers can do this at all, the trail leads back to transformer attention and the geometry of high-dimensional representations.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/in-context-learning/Content licensed under CC BY-NC 4.0.