Chain-of-Thought Reasoning

When you ask a language model a hard arithmetic question — say, "If a store sells 17 apples at $0.85 each and takes a 12% discount, what is the total?" — one thing separates a reliable answer from a wrong one: whether the model shows its work.

The idea is called chain-of-thought (CoT) prompting, and it is one of the most striking discoveries in modern AI. Popularized by a 2022 Google Brain paper by Jason Wei and colleagues, it showed that simply asking a model to reason step-by-step — or including a few worked examples in the prompt — dramatically boosts accuracy on problems involving arithmetic, logic, and symbolic reasoning.

The intuition is old and human: scratch paper exists because our working memory is finite. A language model's "working memory" is the context window, and intermediate tokens are the scratch paper. Each reasoning step produces a token that the next step can attend to — the model is literally computing by writing.

The demo below simulates how a language model approaches a multi-step arithmetic problem in two modes: One-Shot (blurt the answer directly) and Step-by-Step (chain-of-thought). Click a mode, then hit Run to see each approach unfold.

<!-- {{c_html_desc}} -->
<div class="cot-app">
  <p class="hint">{{hint_text}}</p>

  <div class="problem-box">
    <label class="problem-label">{{problem_label}}</label>
    <div class="problem-text" id="problem-text"></div>
  </div>

  <div class="mode-btns">
    <button id="btn-oneshot" type="button" class="mode-btn active" data-mode="oneshot">{{btn_oneshot}}</button>
    <button id="btn-cot" type="button" class="mode-btn" data-mode="cot">{{btn_cot}}</button>
  </div>

  <div class="action-row">
    <button id="btn-run" type="button" class="run-btn">{{btn_run}}</button>
    <button id="btn-new" type="button" class="ghost-btn">{{btn_new}}</button>
  </div>

  <div class="output-box" id="output-box">
    <div class="output-placeholder" id="output-placeholder">{{output_placeholder}}</div>
    <div id="steps-container" class="steps-container" style="display:none"></div>
  </div>

  <div class="result-row" id="result-row" style="display:none">
    <span class="result-label">{{result_label}}</span>
    <span class="result-value" id="result-value"></span>
    <span class="result-check" id="result-check"></span>
  </div>
</div>

/* {{c_css_desc}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }

.cot-app { max-width: 560px; margin: 0 auto; padding: .5rem 0; }

.hint { font-size: .88rem; color: #555; margin: 0 0 .9rem; line-height: 1.5; }

.problem-box { background: #f0f4f8; border: 1px solid #d1dce8; border-radius: 10px;
               padding: .7rem 1rem; margin-bottom: .85rem; }
.problem-label { font-size: .75rem; font-weight: 700; text-transform: uppercase;
                 letter-spacing: .04em; color: #6b8296; display: block; margin-bottom: .3rem; }
.problem-text { font-size: .97rem; color: #1d3557; line-height: 1.5; font-weight: 500; }

.mode-btns { display: flex; gap: .5rem; margin-bottom: .75rem; }
.mode-btn { flex: 1; padding: .45rem .6rem; font: 600 .88rem system-ui, sans-serif;
            border: 2px solid #1d3557; border-radius: 8px; cursor: pointer;
            background: #fff; color: #1d3557; transition: all .15s; }
.mode-btn.active { background: #1d3557; color: #fff; }

.action-row { display: flex; gap: .5rem; margin-bottom: .85rem; }
.run-btn { flex: 2; font: 700 .95rem system-ui, sans-serif; padding: .5rem 1rem;
           background: #2a7d4f; color: #fff; border: none; border-radius: 8px; cursor: pointer; }
.run-btn:hover { background: #235f3c; }
.ghost-btn { flex: 1; font: 600 .88rem system-ui, sans-serif; padding: .5rem .8rem;
             background: #fff; color: #1d3557; border: 1px solid #1d3557; border-radius: 8px; cursor: pointer; }

.output-box { min-height: 140px; background: #fafbfc; border: 1px solid #dce3ea;
              border-radius: 10px; padding: .75rem 1rem; margin-bottom: .7rem; position: relative; }
.output-placeholder { color: #aab; font-size: .9rem; font-style: italic; }

.steps-container { display: flex; flex-direction: column; gap: .5rem; }
.step { display: flex; gap: .6rem; align-items: flex-start; opacity: 0;
        animation: fadeIn .3s forwards; }
.step-num { min-width: 22px; height: 22px; border-radius: 50%; background: #1d3557;
            color: #fff; font: 700 .75rem system-ui; display: flex; align-items: center;
            justify-content: center; margin-top: .1rem; flex-shrink: 0; }
.step-text { font-size: .9rem; line-height: 1.5; color: #1d3557; }
.step.direct .step-num { background: #888; }
.step.direct .step-text { color: #444; }

.result-row { display: flex; align-items: center; gap: .6rem; padding: .5rem .8rem;
              border-radius: 8px; background: #eef6f0; border: 1px solid #b8dcc5; }
.result-label { font-size: .85rem; color: #555; }
.result-value { font: 700 1rem system-ui; color: #1d3557; }
.result-check { font-size: 1rem; }
.result-row.wrong { background: #fdf0f0; border-color: #e8b8b8; }

@keyframes fadeIn { to { opacity: 1; } }

// Code not found

Notice the pattern: one-shot sometimes gets lucky on easy problems, but as the problem grows harder the chain-of-thought mode makes each sub-result visible and checkable. The intermediate tokens are not decoration — they are the computation. Without them, the model must compress the entire reasoning into a single forward pass through the output head.

Why does writing intermediate steps help so much? The answer is about computational depth.

A transformer block is a fixed-depth circuit: each token passes through the same number of layers regardless of problem difficulty. If you ask for the answer in one step, every sub-calculation must be compressed into that single output position — the model cannot reuse earlier results because they were never written down.

Chain-of-thought breaks that ceiling. Each intermediate token becomes a new input to the next generation step, effectively adding more sequential computation layers. The total depth of the computation is no longer bounded by the transformer's fixed number of layers; it is bounded by the length of the reasoning trace.

Formally, CoT lets a constant-depth transformer simulate a much deeper — potentially polynomial-depth — circuit, because each token generation is a new forward pass. Research by Merrill & Sabharwal (2023) showed that with CoT, transformers can solve problems in $\text{TC}^1$ that they cannot solve in $\text{TC}^0$ without scratch space. In plain terms: hard problems need more reasoning steps, and those steps must be visible.

Unlike NP-complete problems, which are hard because verification requires exponential search, CoT's gains are about depth not search — and that makes them practically achievable just by changing the prompt.

Chain-of-thought is not an academic curiosity — it is the engine inside nearly every AI system that needs to be reliably correct:

• Math and science tutoring: step-by-step solutions let students follow and catch errors, not just copy a number.
• Code generation: writing pseudocode or a plan before generating code mirrors CoT exactly — the model debugs its own logic before committing to syntax.
• Medical and legal reasoning: high-stakes decisions require an auditable trace; a bare answer with no justification is not trusted or safe.
• Multi-hop question answering: answering "Which country has a higher GDP per capita, the one that borders France to the east and also borders Switzerland?" requires chaining facts — one-shot answers routinely fail.
• Transformers and attention: the attention mechanism that reads the context is the same mechanism that reads the CoT trace — understanding one deepens the other.

The flip side: CoT costs tokens. Longer reasoning traces mean higher latency and more compute. Current research explores compressed scratchpads, process reward models that score reasoning quality, and tree-of-thought extensions that branch and prune candidate reasoning paths.

Chain-of-thought prompting revealed something profound: the tokens a model generates are not just its answer — they are its working memory. By making intermediate steps explicit, a fixed-depth architecture gains the ability to tackle problems that otherwise exceed its reach.

The practical upshot is elegant: if you want a language model to reason reliably, give it room to think out loud. The scratchpad is not a crutch — it is the computation. And the next time an AI gives you a confident wrong answer on a multi-step problem, ask it to show its work. You may be surprised how much the answer changes.