Introduction

Imagine teaching a dog to fetch — not by explaining the rules, but purely by giving treats when it does well and ignoring it when it doesn't. After enough runs, the dog builds an internal sense of which actions lead to treats. Q-learning is that idea, made precise and proved to work.

Introduced by Christopher Watkins in his 1989 PhD thesis and formally proved convergent by Watkins and Peter Dayan in 1992, Q-learning is a model-free reinforcement learning algorithm. "Model-free" means the agent does not need a map of the environment — it just takes actions, observes what reward it gets, and updates its beliefs. Given enough time, those beliefs converge to the optimal policy: the best possible action in every situation.

The core idea is deceptively simple. For every (state, action) pair the agent maintains a number called a Q-value — short for quality — that estimates how much total future reward you can expect if you take that action right now and then play optimally forever after. At every step the agent improves its estimates using the Bellman equation, a one-line formula that ties today's value to tomorrow's best outcome.

That single equation, iterated millions of times, is how a program learned to play 49 Atari games at superhuman level in 2015 (Deep Q-Network, Mnih et al., Nature). It remains the conceptual engine behind almost every major reinforcement learning breakthrough since.

Train an Agent

Below is a 4 × 4 grid world. The agent (blue) starts in the top-left corner and must reach the goal (green star) while avoiding the pit (red). It receives +10 for reaching the goal, −10 for falling in the pit, and −1 for every other step.

Press Step to run one episode manually, or Train 50 to watch 50 episodes at once. The numbers in each cell show the current best Q-value from that state — they start at zero and converge toward the true optimal values as experience accumulates.

<div class="controls">
  <button id="btn-step" type="button">Step (1 episode)</button>
  <button id="btn-train" type="button">Train 50</button>
  <button id="btn-reset" type="button" class="ghost">Reset</button>
  <span class="ep-label">Episode: <b id="ep-count">0</b></span>
</div>
<div class="grid-wrap">
  <div id="grid" class="grid"></div>
</div>
<div class="legend">
  <span class="leg-item"><span class="swatch agent"></span> Agent</span>
  <span class="leg-item"><span class="swatch goal"></span> Goal (+10)</span>
  <span class="leg-item"><span class="swatch pit"></span> Pit (−10)</span>
  <span class="leg-item"><span class="swatch normal"></span> Step (−1)</span>
</div>
<div class="info">Numbers show best Q-value per cell. Brighter = higher value learned.</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
.controls { display: flex; align-items: center; gap: .5rem; flex-wrap: wrap; margin-bottom: .7rem; }
button { font: 600 13px system-ui; padding: .4rem .8rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
button:disabled { opacity: .45; cursor: default; }
.ep-label { font-size: .9rem; color: #555; margin-left: .3rem; }
.grid-wrap { display: flex; justify-content: center; }
.grid { display: grid; grid-template-columns: repeat(4, 80px); gap: 4px; }
.cell { width: 80px; height: 80px; border-radius: 8px; display: flex; flex-direction: column;
        align-items: center; justify-content: center; position: relative;
        font-size: .72rem; font-weight: 600; border: 2px solid transparent; transition: background .25s; }
.cell.normal { background: #d6dde4; color: #333; }
.cell.goal   { background: #2d9e6e; color: #fff; }
.cell.pit    { background: #c0392b; color: #fff; }
.cell.agent  { border-color: #1d6fa5; }
.cell .qval  { font-size: .78rem; font-weight: 700; }
.cell .icon  { font-size: 1.4rem; line-height: 1; }
.legend { display: flex; gap: .9rem; flex-wrap: wrap; margin: .6rem 0 .2rem; font-size: .8rem; }
.leg-item { display: flex; align-items: center; gap: .3rem; }
.swatch { width: 14px; height: 14px; border-radius: 3px; border: 1.5px solid #888; }
.swatch.agent  { background: #d6dde4; border-color: #1d6fa5; }
.swatch.goal   { background: #2d9e6e; border: none; }
.swatch.pit    { background: #c0392b; border: none; }
.swatch.normal { background: #d6dde4; border: none; }
.info { font-size: .78rem; color: #666; margin-top: .2rem; }

// Q-learning on a 4x4 grid world.
// States: 0..15 (row-major). Actions: 0=up 1=down 2=left 3=right
// Layout:
//   S . . .    (S=start=0, .=normal)
//   . . . .
//   . . P .    (P=pit=10)
//   . . . G    (G=goal=15)

const ROWS = 4, COLS = 4, N = 16;
const GOAL = 15, PIT = 10, START = 0;
const ACTIONS = 4; // up down left right
const DR = [-1, 1, 0, 0];
const DC = [ 0, 0,-1, 1];

// Hyperparameters
const ALPHA = 0.1;   // learning rate
const GAMMA = 0.9;   // discount
const EPS_START = 1.0;
const EPS_END   = 0.05;
const EPS_DECAY = 0.97; // per episode

let Q, eps, totalEp;

function initQ() {
  Q = Array.from({length: N}, () => new Float64Array(ACTIONS));
  eps = EPS_START;
  totalEp = 0;
}
initQ();

function stateOf(r, c) { return r * COLS + c; }
function rowOf(s)  { return Math.floor(s / COLS); }
function colOf(s)  { return s % COLS; }
function isTerminal(s) { return s === GOAL || s === PIT; }

function step(s, a) {
  const nr = rowOf(s) + DR[a];
  const nc = colOf(s) + DC[a];
  if (nr < 0 || nr >= ROWS || nc < 0 || nc >= COLS) {
    // Bumps into wall — stay
    const r = -1;
    return { ns: s, r };
  }
  const ns = stateOf(nr, nc);
  const r = ns === GOAL ? 10 : ns === PIT ? -10 : -1;
  return { ns, r };
}

function maxQ(s) {
  let m = Q[s][0];
  for (let a = 1; a < ACTIONS; a++) if (Q[s][a] > m) m = Q[s][a];
  return m;
}
function greedyAction(s) {
  let best = 0, bv = Q[s][0];
  for (let a = 1; a < ACTIONS; a++) if (Q[s][a] > bv) { bv = Q[s][a]; best = a; }
  return best;
}

function runEpisode() {
  let s = START;
  let steps = 0;
  while (!isTerminal(s) && steps < 200) {
    // epsilon-greedy action
    const a = Math.random() < eps ? Math.floor(Math.random() * ACTIONS) : greedyAction(s);
    const { ns, r } = step(s, a);
    // Bellman update
    const td = r + (isTerminal(ns) ? 0 : GAMMA * maxQ(ns)) - Q[s][a];
    Q[s][a] += ALPHA * td;
    s = ns;
    steps++;
  }
  eps = Math.max(EPS_END, eps * EPS_DECAY);
  totalEp++;
}

// Animate one episode step-by-step and return a Promise
function runEpisodeAnimated() {
  return new Promise(resolve => {
    let s = START;
    let steps = 0;
    function tick() {
      if (isTerminal(s) || steps >= 200) { resolve(); return; }
      const a = Math.random() < eps ? Math.floor(Math.random() * ACTIONS) : greedyAction(s);
      const { ns, r } = step(s, a);
      const td = r + (isTerminal(ns) ? 0 : GAMMA * maxQ(ns)) - Q[s][a];
      Q[s][a] += ALPHA * td;
      s = ns; steps++;
      render(s);
      requestAnimationFrame(tick);
    }
    tick();
  });
}

// ---- Render ----
function cellColor(s) {
  if (s === GOAL) return null;
  if (s === PIT)  return null;
  const v = maxQ(s); // ranges roughly -10 .. +10
  const t = Math.max(0, Math.min(1, (v + 10) / 20));
  const r = Math.round(180 - t * 80);
  const g = Math.round(190 - t * 40 + t * 60);
  const b = Math.round(200 - t * 60);
  return `rgb(${r},${g},${b})`;
}

function render(agentState) {
  const grid = document.getElementById('grid');
  grid.innerHTML = '';
  for (let s = 0; s < N; s++) {
    const div = document.createElement('div');
    let cls = 'cell ';
    if (s === GOAL) cls += 'goal';
    else if (s === PIT) cls += 'pit';
    else cls += 'normal';
    if (s === agentState) cls += ' agent';
    div.className = cls;

    const icon = document.createElement('span');
    icon.className = 'icon';
    if (s === GOAL) icon.textContent = '★';
    else if (s === PIT) icon.textContent = '✕';
    else if (s === agentState) icon.textContent = '●';
    else icon.textContent = '';
    div.appendChild(icon);

    if (s !== GOAL && s !== PIT) {
      const qv = document.createElement('span');
      qv.className = 'qval';
      qv.textContent = maxQ(s).toFixed(1);
      div.appendChild(qv);
      div.style.background = cellColor(s);
    }
    grid.appendChild(div);
  }
  document.getElementById('ep-count').textContent = totalEp;
}

render(START);

let busy = false;

document.getElementById('btn-step').addEventListener('click', async () => {
  if (busy) return;
  busy = true;
  setBtns(true);
  eps = Math.max(eps, 0.2); // keep some exploration for animation
  await runEpisodeAnimated();
  eps = Math.max(EPS_END, eps * EPS_DECAY);
  totalEp++;
  render(START);
  busy = false;
  setBtns(false);
});

document.getElementById('btn-train').addEventListener('click', () => {
  if (busy) return;
  for (let i = 0; i < 50; i++) runEpisode();
  render(START);
});

document.getElementById('btn-reset').addEventListener('click', () => {
  if (busy) return;
  initQ();
  render(START);
});

function setBtns(dis) {
  document.getElementById('btn-step').disabled = dis;
  document.getElementById('btn-train').disabled = dis;
  document.getElementById('btn-reset').disabled = dis;
}
Result
Console

Notice how the Q-values spread backwards from the goal: the cell next to the goal learns first, then its neighbours, then theirs — exactly the backward induction that the Bellman equation encodes. After enough episodes the agent navigates a perfect path every time. Compare the neural-network training article to see how the same principle scales to millions of parameters.

The Bellman Equation

Q-learning's update rule is a single line derived from the Bellman optimality equation:

Q(s,a)←Q(s,a)+α⋅[r+γ⋅max⁡a′Q(s′,a′)−Q(s,a)]Q(s, a) \leftarrow Q(s, a) + \alpha \cdot [r + \gamma \cdot \max_{a'} Q(s', a') - Q(s, a)]

Every symbol does a job:

  • Q(s, a) — current estimate of how good action a is in state s
  • α\alpha — learning rate (how fast beliefs update; too high and the agent forgets, too low and it learns slowly)
  • r — the reward just received
  • γ\gamma — discount factor (values between 0 and 1 shrink distant rewards; γ=0\gamma = 0 means "live in the present", γ=1\gamma = 1 means "plan forever")
  • max⁡a′Q(s′,a′)\max_{a'} Q(s', a') — the best estimated value of the next state s′s' across all actions
  • The term in brackets is the temporal-difference (TD) error: how surprised you were by this step

Watkins and Dayan (1992) proved that if every (state, action) pair is visited infinitely often and the learning rate decays appropriately, Q-values converge to their optimal values with probability 1. The proof is a stochastic approximation argument — the TD errors form a contraction that drives the table toward a fixed point.

Where it breaks down. For problems with millions or billions of states (pixels, sensor readings) a table is infeasible. The Deep Q-Network (DQN) of Mnih et al. replaces the table with a neural network, enabling superhuman Atari play — but the convergence guarantee disappears. Non-linear function approximation can diverge ("deadly triad"), so modern deep RL adds experience replay, target networks, and many other stabilizing tricks. The algorithm remains classified as solved in complexity terms: optimal tabular Q-learning runs in polynomial time per step, and the full convergence happens in expected polynomial episodes for finite MDPs.

Where It Matters

Q-learning is not a curiosity — it is the conceptual seed behind some of the most dramatic AI achievements of the past decade:

  • Game playing: DQN (2015) reached superhuman performance on 49 Atari games from raw pixels. AlphaGo and AlphaZero extended the idea to Go and Chess using a hybrid of Q-learning and tree search.
  • Robotics: robots learn to walk, grasp, and assemble parts through simulated trial and error — millions of virtual episodes that would be impossible to collect physically.
  • Traffic and logistics: traffic-light timing, order routing in warehouses, and ride-sharing dispatch are all sequential decision problems where RL agents outperform hand-tuned rules.
  • Data-centre cooling: DeepMind used RL to cut Google data-centre cooling energy by ~40 %, identifying non-obvious control strategies that human engineers had missed.
  • Drug discovery and materials science: RL agents propose molecular structures, run virtual tests, and iterate — compressing years of lab work into days of computation.
  • Dialogue systems: chatbots and recommendation engines use RL to optimise multi-turn conversations for user satisfaction rather than just the next response.

The common thread: a problem with states, actions, delayed rewards, and no reliable forward model. Anywhere that description fits, the Bellman equation is worth trying. Pair Q-learning with a neural network and the range of solvable problems explodes.

Conclusion

Q-learning is one of those rare algorithms whose elegance matches its impact. A single equation — update your estimate of today's value toward the discounted best value you see tomorrow — is enough to discover optimal strategies in any finite environment, with no model of the world whatsoever.

Watkins proved it works; DQN proved it scales. The same Bellman backbone now lives inside AlphaZero, robotics simulators, data-centre controllers, and drug-design pipelines. Each new application is really asking the same question the grid-world agent asks: given where I am and what I just experienced, what is the best thing to do next?

The answer, iterated a few million times, turns out to be remarkably good — and that is the beautiful, slightly unsettling promise of reinforcement learning.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/q-learning/Content licensed under CC BY-NC 4.0.