In-Context Learning

You hand a language model three examples — "cat → feline, dog → canine, eagle → ?" — and it completes the pattern. No training loop ran. No gradient updated a single weight. The model never even saw this mapping during pretraining. Yet it gets it right.

This is in-context learning (ICL): the ability of a large language model to adapt to a new task at inference time, guided only by examples embedded in the prompt. The weights are frozen; all the "learning" happens inside the context window.

ICL is strange enough to deserve a closer look. Classical machine learning separates training from inference sharply: you train once, then you predict. ICL blurs that line. The same forward pass that produces the answer is also, somehow, the act of picking up the pattern. Understanding why that is possible turns out to reveal something deep about what transformer attention is actually doing.

Each row below is one example the model sees before your query. Add or remove examples, change the rule, then hit Predict — the simulated model will complete the pattern using only what is in the context.

<!-- {{c_icl_demo}} -->
<p class="hint">{{hint_para}}</p>
<div id="examples-container">
  <div class="section-label">{{label_examples}}</div>
  <div id="examples-list"></div>
  <button id="add-example" type="button" class="ghost">{{btn_add_example}}</button>
</div>
<div class="query-row">
  <div class="section-label">{{label_query}}</div>
  <div class="query-line">
    <input id="query-input" type="text" placeholder="{{placeholder_query}}" autocomplete="off" />
    <span class="arrow">→</span>
    <span id="prediction" class="prediction-box">{{prediction_empty}}</span>
  </div>
</div>
<div class="btns">
  <button id="predict-btn" type="button">{{btn_predict}}</button>
  <button id="reset-btn" type="button" class="ghost">{{btn_reset}}</button>
</div>
<div id="status" class="status"></div>

/* {{c_styles}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .8rem; line-height: 1.5; }
.section-label { font-size: .75rem; font-weight: 700; text-transform: uppercase;
                 letter-spacing: .05em; color: #5a7088; margin-bottom: .35rem; }
#examples-container { background: #f4f7fa; border-radius: 10px; padding: .7rem .85rem .5rem;
                      margin-bottom: .7rem; }
#examples-list { display: flex; flex-direction: column; gap: .35rem; margin-bottom: .45rem; }
.example-row { display: flex; align-items: center; gap: .4rem; }
.example-row input { flex: 1; padding: .32rem .55rem; border: 1px solid #cdd9e3; border-radius: 6px;
                     font: 14px ui-monospace, monospace; background: #fff; }
.example-row .arrow { color: #5a7088; font-size: 1rem; }
.example-row .del-btn { background: none; border: none; color: #c92f3c; font-size: 1.1rem;
                        cursor: pointer; padding: 0 .2rem; line-height: 1; }
.query-row { margin-bottom: .7rem; }
.query-line { display: flex; align-items: center; gap: .5rem; }
.query-line input { flex: 1; padding: .38rem .6rem; border: 1.5px solid #1d3557; border-radius: 6px;
                    font: 14px ui-monospace, monospace; background: #fff; }
.arrow { color: #5a7088; font-size: 1rem; }
.prediction-box { flex: 1; min-height: 2rem; display: flex; align-items: center;
                  padding: .32rem .6rem; background: #e8eef3; border-radius: 6px;
                  font: 600 14px ui-monospace, monospace; color: #1d3557; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin-bottom: .4rem; }
button { font: 600 14px system-ui, sans-serif; padding: .42rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.status { font-size: .9rem; font-weight: 600; min-height: 1.3em; color: #0a7d33; }
.status.bad { color: #c92f3c; }

// Code not found

Notice that changing even one example rewires the answer immediately. The "model" here uses simple frequency counting — a toy version of what real induction heads do when they copy patterns from earlier in the context. What is striking in both cases is that no parameter was touched: the behavior changed purely because the context changed.

How does in-context learning work mechanistically? The leading explanation centers on induction heads — a specific two-layer attention circuit described by Olsson et al. (2022).

• What an induction head does. Layer 1 builds a map of which token follows which. Layer 2 looks up the current token in that map and copies whatever came after its most recent occurrence. In sequence A B … A → ?, the head predicts B.
• Why that's enough for ICL. Given few-shot examples input₁ → output₁, input₂ → output₂, … inputₙ → ?, the induction circuit matches inputₙ to the earlier occurrences of similar inputs and copies their following output tokens.
• The gradient-descent connection. Akyürek et al. (2022) showed that for linear tasks, a transformer running ICL implicitly implements a form of gradient descent in activation space — the forward pass is a learning step.
• It is not fine-tuning. ICL produces no persistent change: start a new conversation without the examples and the "learning" vanishes. The model has not been updated; it has been steered.
• Limits. ICL breaks when the required mapping conflicts strongly with pretraining priors, when the context window is too short, or when the task demands multi-step reasoning that exceeds what a fixed-depth forward pass can implicitly compute.

The phenomenon is well-understood at the mechanistic level (Olsson et al., 2022; Akyürek et al., 2022), though questions about its precise sample-efficiency bounds and failure modes remain active research.

In-context learning is the engine behind almost everything interesting that large language models do at deployment time:

• Prompt engineering: every few-shot prompt is an ICL invocation. Curating better examples can flip accuracy from 50% to 90% without touching a model parameter.
• Chain-of-thought reasoning: showing the model how to reason through solved examples before posing a hard problem is ICL applied to reasoning steps, not just answers.
• Classification without a head: add labeled examples to the context and the model classifies new inputs without any fine-tuned softmax layer.
• Tool-use and agents: giving an agent examples of how to invoke a tool in the prompt is ICL applied to action sequences — the same pattern-copying mechanism governs when to call a function.
• Rapid prototyping: teams ship new behaviors in hours by writing examples, not in weeks by curating datasets and running training jobs.

ICL also reveals the limits of prompt-only adaptation. It cannot inject new factual knowledge (the weights have not changed), and its "learning" evaporates when the session ends. For durable or high-accuracy tasks, transformer fine-tuning or retrieval augmentation is still the right tool.

In-context learning is one of the most surprising properties to emerge from scaling language models. The same network that was trained on next-token prediction turns out to harbour a gradient-descent-like learning algorithm inside its attention heads — one that activates whenever it sees examples in the context, and silently copies patterns that fit.

The mechanism is real, the circuit is identified, and the applications are everywhere. But remember the boundary: ICL steers a frozen model; it does not change it. The moment the prompt window closes, the "learning" disappears. For everything that needs to persist, you still need the optimizer — and for the deep theory of why transformers can do this at all, the trail leads back to transformer attention and the geometry of high-dimensional representations.