Recurrent Neural Networks

A standard neural network looks at one input and produces one output. Show it the word bank, and it answers based on that word alone — context be damned. A recurrent neural network (RNN) does something cleverer: it routes its own output back to itself as an additional input. That loop threads a hidden state — a small vector of numbers — through every step of the sequence, so each new token is interpreted in the light of everything that came before.

The idea is old. In 1982 John Hopfield described feedback networks for associative memory; by the late 1980s researchers were training RNNs on sequences with backpropagation through time (BPTT), unrolling the loop step by step into a very deep network, then running gradient descent backwards. The architecture unlocked speech recognition, handwriting generation and language modelling — tasks where order matters and context stretches across many tokens.

The catch arrived quickly. Gradients flowing backwards through dozens or hundreds of time steps tend to shrink exponentially — the vanishing-gradient problem — or explode — the exploding-gradient problem. Either way, the network loses its grip on events far back in the sequence. That single flaw has shaped almost every advance in sequence modelling since.

The demo below runs a small Elman RNN with two hidden units through a hand-crafted sequence of five tokens. Each token is represented as a single number (its index). At each step the network computes a new hidden state using its recurrence formula and displays both the input and the evolving state vector.

<p class="hint">{{hint}}</p>
<div class="controls">
  <label>{{seq_label}}
    <input type="range" id="seqLen" min="3" max="12" value="5">
    <span id="seqLenVal">5</span>
  </label>
  <button id="stepBtn" type="button">{{step_btn}}</button>
  <button id="resetBtn" type="button" class="ghost">{{reset_btn}}</button>
</div>
<div id="timeline"></div>
<div class="legend">
  <span class="dot input"></span> {{legend_input}} &nbsp;
  <span class="dot h0"></span> h[0] &nbsp;
  <span class="dot h1"></span> h[1]
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; font-size: 14px; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
.controls { display: flex; align-items: center; gap: .8rem; flex-wrap: wrap; margin-bottom: .8rem; }
label { display: flex; align-items: center; gap: .4rem; font-size: .88rem; }
input[type=range] { width: 100px; accent-color: #1d3557; }
button { font: 600 13px system-ui; padding: .35rem .8rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
#timeline { display: flex; gap: 6px; flex-wrap: wrap; align-items: flex-end; min-height: 140px; }
.step-col { display: flex; flex-direction: column; align-items: center; gap: 4px; width: 52px; }
.step-col .label { font-size: .72rem; color: #666; }
.bar-wrap { display: flex; flex-direction: column; align-items: center; gap: 3px; width: 100%; }
.bar { width: 28px; border-radius: 4px; transition: height .35s ease; min-height: 4px; }
.bar.input  { background: #457b9d; }
.bar.h0     { background: #e63946; }
.bar.h1     { background: #2a9d8f; }
.bar-val { font-size: .7rem; color: #333; text-align: center; }
.legend { margin-top: .6rem; font-size: .8rem; display: flex; align-items: center; gap: 4px; }
.dot { width: 10px; height: 10px; border-radius: 50%; display: inline-block; }
.dot.input { background: #457b9d; }
.dot.h0    { background: #e63946; }
.dot.h1    { background: #2a9d8f; }

// Code not found

Watch the hidden state columns: earlier tokens leave a trace that shapes later computations. Then notice what happens as you increase the sequence length — the influence of the very first token fades. That gradual erasure is the vanishing-gradient problem made visible.

Training an RNN means computing gradients through the unrolled recurrence — effectively through a very deep network with shared weights. That causes two symmetric disasters:

• Vanishing gradients: if the weight matrix has spectral radius less than 1, each timestep multiplies the gradient by a number smaller than 1. After 100 steps the signal is ×(0.9)¹⁰⁰ ≈ 0.000027 of its original size — indistinguishable from noise. The network simply cannot attribute an outcome to a cause that happened early in the sequence.
• Exploding gradients: if the spectral radius exceeds 1, gradients blow up exponentially. The fix — gradient clipping — caps the norm of the gradient vector before each update. It is a workaround, not a solution.

The solution to vanishing gradients was gating. In 1997 Sepp Hochreiter and Jürgen Schmidhuber introduced the Long Short-Term Memory (LSTM), which adds three learned gates — input, forget and output — that directly control what enters, what stays, and what leaves the hidden state. A cell state runs forward mostly unchanged, like a conveyor belt, and gradients can flow back through it without collapsing. In 2014 Kyunghyun Cho et al. proposed the simpler Gated Recurrent Unit (GRU), merging the forget and input gates into one reset gate and one update gate.

Even with gating, plain recurrent models struggle beyond a few hundred steps. Modern sequence modelling has largely moved to Transformers (attention over all positions in parallel) and state-space models (structured recurrences that avoid gradient issues by design). But the original RNN remains the clearest illustration of why sequence modelling is hard — and why the solution required rethinking how networks carry memory.

For more on the learning complexity that sits behind all neural training see P vs NP and non-convex optimization.

Despite being largely superseded by Transformers for language, RNNs and their gated variants remain important across many domains:

• Speech recognition: LSTMs were the backbone of Google's voice search and Apple's Siri for years, processing audio frames one by one while accumulating phoneme context.
• Machine translation: the encoder-decoder LSTM architecture (Sutskever et al., 2014) was the first neural system to approach human translation quality on short sentences, and spawned the attention mechanism that later became Transformers.
• Time-series forecasting: financial returns, sensor readings and energy consumption all arrive as sequences where the history genuinely matters. LSTMs and GRUs remain competitive here, especially on irregular or short series where Transformers over-fit.
• Music and handwriting generation: RNNs trained character-by-character or note-by-note can generate plausible continuations of human style, as shown in Graves's 2013 handwriting synthesis work.
• Bioinformatics: protein secondary structure prediction and DNA sequence analysis naturally map to sequence-to-sequence tasks where recurrent models excel.

The conceptual legacy is even broader. The idea that a model should carry an explicit memory state forward through time, updated at each step by new evidence, reappears in reinforcement learning agents, world models and even the modern selective-state-space models (Mamba, 2023) that are challenging Transformer dominance on long sequences.

The recurrent neural network made a bold promise: feed the output back into the input and the network will remember its own past. For short sequences that promise holds — the hidden state carries genuine context. For long sequences the vanishing-gradient problem quietly empties that memory, step by exponential step.

The fixes — LSTM gates, GRU resets, attention, Transformers, state-space models — are each a different answer to the same question: how do you keep a useful signal alive across hundreds or thousands of steps? RNNs were the first serious attempt, and understanding why they fail is still the best entry point into why sequence modelling is hard.

The next time an autocomplete system finishes your sentence, or a voice assistant catches your meaning across a long pause, there is a good chance that somewhere in the stack a hidden state is quietly accumulating the past — and a gating mechanism is deciding, step by step, what to remember and what to let go.