Reinforcement Learning

Most machine learning needs a teacher: show the model a photo, tell it "cat", repeat a million times. Reinforcement learning throws that away. There is no answer key. There is only an agent moving through a world, and once in a while a number — the reward — that says "that was good" or "that was bad."

Think of a robot in a room, a player in a game, a thermostat in a house. Each picks an action, the world changes, and a reward trickles in. The catch: the reward often comes much later than the action that earned it. Reach the goal and you score — but which of the hundred earlier moves deserves the credit?

That is the whole problem. Out of delayed, sparse rewards, the agent must work out a policy: a rule that says, in every situation, what to do next. And the way it does so turns out to be a beautifully simple idea about letting value flow backward from the future.

Here is a tiny gridworld. The agent starts somewhere and wants to reach the green goal (+1) while avoiding the red traps (−1). It can move up, down, left or right. At the start it knows nothing — every square is grey, value zero.

<p class="hint">{{hint}}</p>
<div id="grid" class="grid"></div>
<div class="status" id="status">{{sweep0}}</div>
<div class="btns">
  <button id="step" type="button">{{btn_step}}</button>
  <button id="run" type="button">{{btn_run}}</button>
  <button id="reset" type="button" class="ghost">{{btn_reset}}</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
.grid { display: grid; grid-template-columns: repeat(5, 58px); gap: 4px; margin: .4rem 0; }
.cell { width: 58px; height: 58px; display: flex; flex-direction: column; align-items: center;
        justify-content: center; border-radius: 8px; border: 1px solid #cdd9e3;
        font: 700 13px ui-monospace, monospace; position: relative; transition: background .25s; }
.cell .arrow { font-size: 18px; line-height: 1; margin-bottom: 1px; }
.cell .val { font-size: 12px; opacity: .85; }
.cell.wall { background: #4a4a4a; border-color: #333; }
.cell.goal { background: #2a9d4a; color: #fff; border-color: #1f7d39; }
.cell.trap { background: #e63946; color: #fff; border-color: #c92f3c; }
.status { font-size: 1rem; font-weight: 600; margin: .5rem 0; min-height: 1.4em; color: #1d3557; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

// Code not found

Press Step to run one sweep of value iteration. Each square asks its neighbours, "what is the best you can promise me?" and updates its own value to the best discounted reward it can reach. Watch the green glow of the goal seep outward, one ring per step, until even the farthest corner knows which way to walk. Then the arrows — the policy — point straight home. No one ever told the agent the route; it fell out of the reward alone.

Underneath, reinforcement learning rests on the Markov Decision Process (MDP): a set of states, the actions available in each, the probability that an action moves you from one state to another, and the reward you collect. The goal is a policy that maximises total reward over time, with future reward shrunk by a discount factor.

• Known model, exact solution. If you know the MDP, the Bellman optimality equation pins down the value of every state, and value iteration or policy iteration (Bellman, 1957; Howard, 1960) converge to the optimal policy. For a fixed discount, policy iteration runs in polynomial time in the number of states and actions — this is a solved problem.
• The curse of dimensionality. The catch is that the number of states explodes: a chessboard, a robot's joints, a screen of pixels. Enumerating them all is hopeless, so real systems approximate the value with neural networks instead of a table.
• Unknown model. When the agent does not know the transition probabilities, it must learn them by acting. Now the cost is measured in samples — how much experience is needed — and the exploration-vs-exploitation tradeoff makes guarantees far subtler.

So the elegant core is genuinely tractable, much like solving a shortest path. The frontier is scale and uncertainty: huge or unknown worlds where exact solutions give way to approximation, and the hardness shifts from computing the answer to gathering enough experience to estimate it.

"Act now, get rewarded later" describes an enormous range of real problems, and reinforcement learning is the toolkit for all of them:

• Games: from Backgammon to Go and Atari, agents trained on self-play reached and then passed human champions, learning strategies no one taught them.
• Robotics and control: walking, grasping, balancing and steering are all "pick actions that keep paying off," learned in simulation and transferred to hardware.
• Operations: datacentre cooling, traffic-light timing, inventory and energy grids are control problems where a good policy saves real money.
• Tuning AI: large language models are polished with reinforcement learning from human feedback, where the reward is how much a person liked the answer.

Understand value iteration and you have met the engine behind all of it — the same backward flow of value powers neural-network training wherever a system must learn to act, not just to label.

Reinforcement learning hides an almost magical idea behind plain arithmetic: hand an agent nothing but a number that arrives late and rarely, and let it discover, all by itself, the best thing to do in every situation. The trick is to let the value of the goal flow backward until each state knows its worth — and from those values a policy simply falls out.

For a small, known world this is a solved, polynomial-time problem. Scale it up — unknown rules, astronomically many states — and the same idea, wrapped in neural networks and clever exploration, is what taught machines to play Go, steer robots and converse. From one delayed reward, a whole way of acting. That is the quiet power of learning by reward.