LSTM and GRU

Sequences are everywhere in computation: a sentence is a sequence of words, a genome is a sequence of bases, a melody is a sequence of notes. The natural tool for sequences is the recurrent neural network (RNN) — a loop that reads one element at a time and passes a hidden state forward, so each step can "remember" what came before.

The trouble is that a plain RNN suffers from the vanishing-gradient problem: as you backpropagate errors across many time steps the gradients shrink exponentially, meaning the network can barely learn from context more than a handful of steps away. Ask it to translate the first word of a long sentence using information from the last, and it has effectively forgotten.

Sepp Hochreiter and Jürgen Schmidhuber introduced the Long Short-Term Memory (LSTM) in 1997 to fix this. Instead of a single hidden state that gets noisily overwritten at each step, an LSTM maintains a dedicated cell state — a kind of conveyor belt of memory — protected by three multiplicative gates:

• Forget gate — decides what fraction of the old memory to erase.
• Input gate — decides how much of a new candidate value to write in.
• Output gate — decides what part of the cell state to expose as the hidden state.

Each gate is a sigmoid (outputting 0–1) applied to a learned linear combination of the current input and the previous hidden state, so the network learns what to remember and what to discard. Gradients now flow almost unchanged through the cell state, bypassing the vanishing-gradient collapse.

In 2014, Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio proposed the Gated Recurrent Unit (GRU), a streamlined alternative with only two gates — a reset gate and an update gate — and no separate cell state. The GRU is faster to train and often matches LSTM quality on shorter sequences. Both architectures powered the neural machine translation revolution and every major sequence-to-sequence task before the arrival of Transformers.

Feed a short numeric sequence into a single LSTM cell and watch the three gates fire at each step. Each gate outputs a value between 0 and 1 — 0 means fully closed (block this), 1 means fully open (pass this through). The cell state is the long-term memory; the hidden state is what the cell exposes to the outside world.

<p class="hint">{{hint}}</p>
<div class="seq-row">
  <label>{{seq_label}}</label>
  <div class="seq-input-row">
    <input id="seqInput" type="text" value="0.9 0.1 0.8 0.2 0.7 0.3" />
    <button id="resetBtn" type="button" class="ghost">{{reset}}</button>
  </div>
</div>
<div class="timeline" id="timeline"></div>
<div class="cell-panel">
  <div class="gates-grid" id="gatesGrid">
    <div class="gate-block" id="gForget"><div class="gate-label">{{gate_forget}} <span class="gate-sym">f</span></div><div class="gate-bar-wrap"><div class="gate-bar" id="barF"></div></div><div class="gate-val" id="valF">—</div></div>
    <div class="gate-block" id="gInput"><div class="gate-label">{{gate_input}} <span class="gate-sym">i</span></div><div class="gate-bar-wrap"><div class="gate-bar" id="barI"></div></div><div class="gate-val" id="valI">—</div></div>
    <div class="gate-block" id="gOutput"><div class="gate-label">{{gate_output}} <span class="gate-sym">o</span></div><div class="gate-bar-wrap"><div class="gate-bar" id="barO"></div></div><div class="gate-val" id="valO">—</div></div>
    <div class="gate-block" id="gCandidate"><div class="gate-label">{{gate_candidate}} <span class="gate-sym">g̃</span></div><div class="gate-bar-wrap two-sided"><div class="gate-bar cand" id="barC"></div></div><div class="gate-val" id="valC">—</div></div>
  </div>
  <div class="state-row">
    <div class="state-block">
      <div class="state-label">{{state_cell}} <span class="gate-sym">C</span></div>
      <div class="state-bar-wrap two-sided"><div class="state-bar" id="barCell"></div></div>
      <div class="state-val" id="valCell">0.00</div>
    </div>
    <div class="state-block">
      <div class="state-label">{{state_hidden}} <span class="gate-sym">h</span></div>
      <div class="state-bar-wrap two-sided"><div class="state-bar" id="barHidden"></div></div>
      <div class="state-val" id="valHidden">0.00</div>
    </div>
  </div>
</div>
<div class="btns">
  <button id="stepBtn" type="button">{{step_btn}}</button>
  <button id="runBtn" type="button">{{run_btn}}</button>
</div>
<div class="status" id="status">{{press_step}}</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
.seq-row { margin-bottom: .6rem; }
.seq-row label { font-size: .82rem; color: #555; display: block; margin-bottom: .3rem; }
.seq-input-row { display: flex; gap: .4rem; }
#seqInput { flex: 1; font: 14px ui-monospace, monospace; padding: .3rem .5rem; border: 1px solid #bbb; border-radius: 6px; }
.timeline { display: flex; gap: 6px; flex-wrap: wrap; margin-bottom: .7rem; min-height: 32px; }
.titem { padding: .22rem .5rem; border-radius: 6px; font: 600 13px ui-monospace, monospace;
         background: #e8eef3; border: 1px solid #cdd9e3; color: #444; transition: all .2s; }
.titem.active { background: #1d3557; color: #fff; border-color: #1d3557; }
.titem.done { background: #d0f0d9; border-color: #5cb87a; color: #1a5c2a; }
.cell-panel { background: #f7f9fb; border: 1px solid #dde5ee; border-radius: 10px; padding: .8rem 1rem; margin-bottom: .6rem; }
.gates-grid { display: grid; grid-template-columns: 1fr 1fr; gap: .6rem .9rem; margin-bottom: .7rem; }
.gate-block { }
.gate-label { font-size: .78rem; color: #555; margin-bottom: .25rem; }
.gate-sym { font-style: italic; font-size: .82rem; }
.gate-bar-wrap { background: #e2e8f0; border-radius: 4px; height: 12px; overflow: hidden; position: relative; }
.gate-bar { height: 100%; border-radius: 4px; transition: width .35s, background .35s; width: 0%; background: #4a90d9; }
.gate-bar.cand { position: absolute; left: 50%; transform: scaleX(1); transform-origin: left; background: #e07b3c; }
.two-sided { }
.gate-val { font: 600 13px ui-monospace, monospace; color: #333; margin-top: .18rem; }
.state-row { display: grid; grid-template-columns: 1fr 1fr; gap: .6rem .9rem; }
.state-block { }
.state-label { font-size: .78rem; color: #555; margin-bottom: .25rem; }
.state-bar-wrap { background: #e2e8f0; border-radius: 4px; height: 14px; overflow: hidden; position: relative; }
.state-bar { height: 100%; border-radius: 4px; transition: width .35s, left .35s, background .35s; position: absolute; background: #5a7088; }
.state-val { font: 700 14px ui-monospace, monospace; color: #1d3557; margin-top: .2rem; }
.btns { display: flex; gap: .5rem; margin-bottom: .4rem; }
button { font: 600 14px system-ui; padding: .42rem .9rem; border: 1px solid #1d3557; background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
button:disabled { opacity: .45; cursor: default; }
.status { font-size: .9rem; font-weight: 600; min-height: 1.3em; color: #1a5c2a; }
.status.info { color: #444; }
.status.done { color: #1d3557; }

// Code not found

Press Step to advance one time step at a time, or Run all to watch the full sequence. Notice how the forget gate can drop the cell state toward zero (forgetting old context) while the input gate simultaneously writes a new value in. The output gate then filters what gets exposed. This interplay is how the cell learns to carry only the signal that matters across many steps.

LSTM and GRU are firmly in the realm of solved engineering — we know how to train them, and they work reliably — but several subtleties are worth understanding:

• Vanishing gradient (fixed by gates). A plain RNN's hidden state is rewritten at every step by a matrix multiply and a squashing nonlinearity (typically tanh). Backpropagating through T steps multiplies the Jacobian T times; if the largest singular value is less than 1, the gradient vanishes; if greater than 1, it explodes. The LSTM cell state bypasses this: the gradient of the cell state with respect to itself is just the forget gate value — a learned scalar close to 1 — so it can flow back across hundreds of steps almost unchanged. This is the key insight of Hochreiter & Schmidhuber (1997).

• Backpropagation through time (BPTT). Training unrolls the recurrence across T steps and differentiates through the entire unrolled graph. Memory usage scales $O(T)$ in the sequence length, which is why very long sequences (thousands of steps) are expensive. Truncated BPTT approximates gradients by cutting the unroll, at the cost of missing very long-range dependencies.

• Parameter count. An LSTM with hidden size h and input size d has 4 weight matrices of shape (h × (h+d)) plus 4 bias vectors — roughly 4h(h+d) parameters. A GRU has 3 sets, so about 3h(h+d). Large h quickly dominates memory.

• Expressiveness. LSTMs are Turing-complete (given unbounded precision and time), but in practice they are limited by finite precision arithmetic and finite context. Empirically they handle dependencies of ~100–300 steps well; beyond that, attention mechanisms outperform them.

• Status: solved technique. The vanishing-gradient problem for moderate sequence lengths is considered solved by gating. The LSTM and GRU are mature, well-understood tools — not open problems in complexity theory, but engineering achievements with a firm theoretical basis.

Gated recurrent cells were the backbone of applied sequence modeling for nearly a decade (roughly 2014–2020) and remain important in many domains:

• Machine translation. The encoder-decoder architecture (two stacked LSTMs with attention) was the dominant approach to translation until Transformers. Google Translate switched to LSTM-based Neural Machine Translation in 2016.
• Speech recognition. Deep bidirectional LSTMs (reading sequences both forward and backward) power systems like Google's speech API and Amazon Alexa's acoustic models.
• Language modeling. Before GPT, LSTM language models (e.g. ELMo, 2018) produced state-of-the-art contextualized word embeddings used across NLP.
• Time-series forecasting. LSTMs handle irregularly sampled sensor data, financial time series, and power-grid demand with strong inductive bias toward sequential patterns.
• Music and handwriting generation. Char-RNN (Andrej Karpathy, 2015) showed that a single character-level LSTM could generate Shakespeare-like text, code, and music notation — popularizing gated RNNs for creative applications.
• Bioinformatics. Sequence models for protein structure and genome annotation used LSTM layers before attention-only architectures took over.

Today Transformers dominate long-sequence tasks because attention scales more favorably and parallelizes better on GPUs. But for edge devices, real-time streaming inference, and tasks with genuinely short context, LSTMs and GRUs remain practical choices — smaller, faster, and no less accurate.

The insight behind LSTM and GRU is deceptively simple: instead of blindly overwriting your memory at every step, learn when to remember and when to forget. Three gates (or two, in the GRU's leaner design) give the network multiplicative control over information flow, letting gradients travel backward across hundreds of steps without vanishing.

That single idea powered machine translation, speech recognition, and language modeling for a decade — until Transformers replaced recurrence with pure attention. But the gating principle lives on: the input, forget, and output gates of an LSTM are conceptual ancestors of the key-query-value attention mechanism. Understanding how gates tame sequences is still the clearest entry point into why modern neural machine translation and language models work at all.