Introduction

Imagine handing a computer thirty photographs of a coffee mug — taken from different angles — and asking it to show you the mug from a viewpoint that no camera ever captured. Before 2020 that was a research problem spanning decades. Then NeRF arrived and made it almost trivial.

Neural Radiance Fields (Mildenhall et al., 2020) represent a scene not as a mesh or a point cloud, but as a continuous function: given any 3-D position (x,y,z)(x, y, z) and a viewing direction (θ,ϕ)(\theta, \phi), a small neural network outputs the color (r,g,b)(r, g, b) and volume density σ\sigma at that point. The network has no explicit geometry — it simply learns to interpolate what the scene looks like from any direction.

To actually produce an image, NeRF fires a ray through each pixel and marches along it, sampling the network at many points and accumulating color using the volume rendering equation — the same physics that describes light passing through fog or smoke. The whole system is trained end-to-end by minimizing the difference between rendered and real photographs.

The result is a compact, differentiable 3-D model that can be queried for any novel view, with no manual segmentation or 3-D scanning required.

Try It: March a Ray

The heart of NeRF is volume rendering: a ray travels through space, and at each sample point the scene's neural network votes on color and opacity. Step through the demo to see how samples accumulate into a final pixel color.

<!-- NeRF ray-marching demo: step through volume rendering sample by sample -->
<div class="controls">
  <label>Scene: <select id="sceneSelect">
    <option value="sphere">Opaque sphere</option>
    <option value="box">Translucent box</option>
    <option value="fog">Thick fog</option>
  </select></label>
</div>
<canvas id="rayCanvas" width="360" height="220" title="Ray marching visualization"></canvas>
<div class="step-row">
  <button id="prevBtn" type="button" disabled>&#8592; Back</button>
  <span id="stepLabel" class="step-label">Step 1 / 8</span>
  <button id="nextBtn" type="button">Next sample &#8594;</button>
</div>
<div id="sampleInfo" class="sample-info">Press Next sample to march the ray through the scene and accumulate color.</div>
<div class="accum-row">
  <span class="accum-label">Pixel so far:</span>
  <canvas id="pixelCanvas" width="36" height="36" title="Accumulated pixel color"></canvas>
  <span id="accumText" class="accum-text">R=0 G=0 B=0 T=1.00</span>
</div>
<button id="resetBtn" type="button" class="ghost">Reset</button>

/* NeRF demo styles */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
canvas { display: block; border-radius: 8px; border: 1px solid #cdd9e3; background: #0a0e14; }
.controls { margin-bottom: .5rem; font-size: .9rem; }
.controls select { font-size: .85rem; padding: .2rem .4rem; border-radius: 4px; border: 1px solid #adb1b8; }
.step-row { display: flex; align-items: center; gap: .6rem; margin: .5rem 0; }
.step-label { font-size: .85rem; color: #555; flex: 1; text-align: center; }
button { font: 600 13px system-ui; padding: .4rem .8rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button:disabled { opacity: .4; cursor: default; }
button.ghost { background: #fff; color: #1d3557; }
.sample-info { font-size: .82rem; color: #333; min-height: 2.8em; padding: .3rem .1rem; line-height: 1.5; }
.accum-row { display: flex; align-items: center; gap: .6rem; margin: .3rem 0; }
.accum-label { font-size: .82rem; color: #555; }
#pixelCanvas { border-radius: 4px; border: 1px solid #adb1b8; }
.accum-text { font: 600 .8rem ui-monospace, monospace; color: #333; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// NeRF volume rendering demo — ray marching in 1D.
// A ray travels from near to far; at each sample the network returns
// density sigma and color (r,g,b). We apply the volume rendering integral.

const NUM_STEPS = 8;
const RAY_COLOR = '#f4c842';
const BG_COLOR  = '#0a0e14';

// Scene definitions — each maps ray parameter t to density and color.
const SCENES = {
  sphere: {
    // Dense sphere at t≈0.55 with soft background.
    density: (t) => {
      const x = 0.3 + t * 0.7;
      const d = Math.sqrt((x - 0.55) ** 2);
      return d < 0.18 ? 8.0 : 0.05;
    },
    color: (t) => {
      const x = 0.3 + t * 0.7;
      const d = Math.sqrt((x - 0.55) ** 2);
      if (d < 0.18) return [0.9, 0.3, 0.2]; // sphere surface color: warm red
      return [0.05, 0.08, 0.15]; // deep space background
    }
  },
  box: {
    // Hard-edged box occupying t in [0.42, 0.72].
    density: (t) => {
      const x = 0.3 + t * 0.7;
      return (x > 0.42 && x < 0.72) ? 6.0 : 0.03;
    },
    color: (t) => {
      const x = 0.3 + t * 0.7;
      return (x > 0.42 && x < 0.72) ? [0.3, 0.7, 0.4] : [0.05, 0.08, 0.15]; // box: green interior, dark background
    }
  },
  fog: {
    // Homogeneous fog — density increases toward far end.
    density: (t) => 0.6 + 1.2 * t,
    color: (t) => {
      const g = 0.4 + 0.5 * t;
      return [g * 0.8, g * 0.9, g]; // fog: cool grey-blue gradient
    }
  }
};

let currentScene = 'sphere';
let step = 0;         // current sample index (0-based)
let T = 1.0;          // transmittance — starts at 1 (full light), decays as ray is blocked
let accumR = 0, accumG = 0, accumB = 0;   // accumulated pixel color components
const dt = 1.0 / NUM_STEPS;               // step size along the ray

function getScene() { return SCENES[currentScene]; }

// Draw the 2-D side view: background scene + ray + sample points.
function drawScene(highlightStep) {
  const cv = document.getElementById('rayCanvas');
  const ctx = cv.getContext('2d');
  const W = cv.width, H = cv.height;
  ctx.clearRect(0, 0, W, H);
  ctx.fillStyle = BG_COLOR;
  ctx.fillRect(0, 0, W, H);

  const scene = getScene();
  // Render a column-by-column preview of the scene density/color.
  for (let px = 0; px < W; px++) {
    const t = px / W;
    const sigma = scene.density(t);
    const [r, g, b] = scene.color(t);
    const alpha = Math.min(1, sigma * 0.15);
    ctx.fillStyle = `rgba(${Math.round(r*255)},${Math.round(g*255)},${Math.round(b*255)},${alpha})`;
    ctx.fillRect(px, 0, 1, H);
  }

  // Draw the horizontal ray.
  const rayY = H / 2;
  ctx.strokeStyle = RAY_COLOR;
  ctx.lineWidth = 2;
  ctx.beginPath(); ctx.moveTo(0, rayY); ctx.lineTo(W, rayY); ctx.stroke();

  // Draw each sample dot; highlight current, fade past ones.
  for (let i = 0; i < NUM_STEPS; i++) {
    const t = (i + 0.5) / NUM_STEPS;
    const px = t * W;
    const sigma = scene.density(t);
    const [r, g, b] = scene.color(t);

    const isHighlight = (i === highlightStep);
    const isPast = (i < highlightStep);

    ctx.beginPath();
    ctx.arc(px, rayY, isHighlight ? 7 : 5, 0, Math.PI * 2);
    if (isHighlight) {
      ctx.fillStyle = `rgb(${Math.round(r*255)},${Math.round(g*255)},${Math.round(b*255)})`;
      ctx.strokeStyle = '#fff';
      ctx.lineWidth = 1.5;
    } else if (isPast) {
      ctx.fillStyle = `rgba(${Math.round(r*255)},${Math.round(g*255)},${Math.round(b*255)},0.5)`;
      ctx.strokeStyle = 'rgba(255,255,255,0.3)';
      ctx.lineWidth = 1;
    } else {
      ctx.fillStyle = 'rgba(200,200,200,0.25)';
      ctx.strokeStyle = 'rgba(200,200,200,0.4)';
      ctx.lineWidth = 1;
    }
    ctx.fill(); ctx.stroke();

    // Show sigma value above dot.
    if (isHighlight || isPast) {
      ctx.fillStyle = isHighlight ? '#fff' : 'rgba(255,255,255,0.5)';
      ctx.font = isHighlight ? '700 11px ui-monospace,monospace' : '500 9px ui-monospace,monospace';
      ctx.textAlign = 'center';
      ctx.fillText('σ=' + sigma.toFixed(1), px, rayY - 12);
    }
  }

  // Near / far labels.
  ctx.fillStyle = 'rgba(255,255,255,0.5)';
  ctx.font = '10px system-ui';
  ctx.textAlign = 'left';
  ctx.fillText('near', 4, H - 4);
  ctx.textAlign = 'right';
  ctx.fillText('far', W - 4, H - 4);
}

function drawPixel(r, g, b) {
  const cv = document.getElementById('pixelCanvas');
  const ctx = cv.getContext('2d');
  ctx.fillStyle = `rgb(${Math.round(r*255)},${Math.round(g*255)},${Math.round(b*255)})`;
  ctx.fillRect(0, 0, cv.width, cv.height);
}

function clamp01(v) { return Math.max(0, Math.min(1, v)); }

// Core volume rendering step: alpha compositing.
function stepForward() {
  if (step >= NUM_STEPS) return;
  const scene = getScene();
  const t = (step + 0.5) / NUM_STEPS;
  const sigma = scene.density(t);
  const [r, g, b] = scene.color(t);

  // Alpha = 1 - exp(-sigma * dt)  [discrete Beer-Lambert]
  const alpha = 1 - Math.exp(-sigma * dt);
  const weight = T * alpha;

  accumR += weight * r;
  accumG += weight * g;
  accumB += weight * b;
  T *= (1 - alpha);  // Transmittance decays: T *= (1 - alpha)

  const info = document.getElementById('sampleInfo');
  info.innerHTML =
    fmt('<b>Sample %v%</b> &nbsp;σ = %sigma%, α = %alpha%, T = %T%, w = %weight%<br>Color contribution: (+%v2%, +%v3%, +%v4%) &nbsp;→ Accumulated: (%accumR%, %accumG%, %accumB%)', { v: step + 1, sigma: sigma.toFixed(2), alpha: alpha.toFixed(3), T: T.toFixed(3), weight: weight.toFixed(3), v2: (weight*r).toFixed(3), v3: (weight*g).toFixed(3), v4: (weight*b).toFixed(3), accumR: accumR.toFixed(3), accumG: accumG.toFixed(3), accumB: accumB.toFixed(3) });

  drawScene(step);
  drawPixel(clamp01(accumR), clamp01(accumG), clamp01(accumB));
  document.getElementById('accumText').textContent =
    `R=${accumR.toFixed(2)} G=${accumG.toFixed(2)} B=${accumB.toFixed(2)} T=${T.toFixed(2)}`;

  step++;
  document.getElementById('stepLabel').textContent = fmt('Step %step% / %NUM_STEPS%', { step: step, NUM_STEPS: NUM_STEPS });
  document.getElementById('prevBtn').disabled = false;
  document.getElementById('nextBtn').disabled = (step >= NUM_STEPS);
}

function stepBack() {
  if (step <= 0) return;
  // Rewind: replay all steps up to (step-1) from scratch.
  step--;
  T = 1.0; accumR = 0; accumG = 0; accumB = 0;
  const scene = getScene();
  for (let i = 0; i < step; i++) {
    const t = (i + 0.5) / NUM_STEPS;
    const sigma = scene.density(t);
    const [r, g, b] = scene.color(t);
    const alpha = 1 - Math.exp(-sigma * dt);
    const weight = T * alpha;
    accumR += weight * r; accumG += weight * g; accumB += weight * b;
    T *= (1 - alpha);
  }

  if (step === 0) {
    document.getElementById('sampleInfo').textContent = 'Press Next sample to march the ray through the scene and accumulate color.';
    drawPixel(0, 0, 0);
    document.getElementById('accumText').textContent = 'R=0.00 G=0.00 B=0.00 T=1.00';
  } else {
    const t = (step - 1 + 0.5) / NUM_STEPS;
    const sigma = scene.density(t);
    const [r, g, b] = scene.color(t);
    const alpha = 1 - Math.exp(-sigma * dt);
    const weight = (T / (1 - alpha + 1e-9)) * alpha; // approx weight for display after rewind
    document.getElementById('sampleInfo').innerHTML =
      fmt('<b>Sample %step%</b> &nbsp;σ = %sigma%, T = %T%<br>Accumulated: (%accumR%, %accumG%, %accumB%)', { step: step, sigma: sigma.toFixed(2), T: T.toFixed(3), accumR: accumR.toFixed(3), accumG: accumG.toFixed(3), accumB: accumB.toFixed(3) });
    drawPixel(clamp01(accumR), clamp01(accumG), clamp01(accumB));
    document.getElementById('accumText').textContent =
      `R=${accumR.toFixed(2)} G=${accumG.toFixed(2)} B=${accumB.toFixed(2)} T=${T.toFixed(2)}`;
  }

  drawScene(step - 1);
  document.getElementById('stepLabel').textContent = fmt('Step %step% / %NUM_STEPS%', { step: step, NUM_STEPS: NUM_STEPS });
  document.getElementById('prevBtn').disabled = (step === 0);
  document.getElementById('nextBtn').disabled = false;
}

function reset() {
  step = 0; T = 1.0; accumR = 0; accumG = 0; accumB = 0;
  document.getElementById('sampleInfo').textContent = 'Press Next sample to march the ray through the scene and accumulate color.';
  document.getElementById('stepLabel').textContent = fmt('Step 1 / %NUM_STEPS%', { NUM_STEPS: NUM_STEPS });
  document.getElementById('prevBtn').disabled = true;
  document.getElementById('nextBtn').disabled = false;
  drawScene(-1);
  drawPixel(0, 0, 0);
  document.getElementById('accumText').textContent = 'R=0.00 G=0.00 B=0.00 T=1.00';
}

document.getElementById('nextBtn').addEventListener('click', stepForward);
document.getElementById('prevBtn').addEventListener('click', stepBack);
document.getElementById('resetBtn').addEventListener('click', reset);
document.getElementById('sceneSelect').addEventListener('change', (e) => {
  currentScene = e.target.value;
  reset();
});

// Initialize demo on load.
reset();
Result
Console

Notice how early opaque samples dominate — once a ray hits dense material, later samples contribute almost nothing. That is why NeRF can represent sharp object boundaries even though the underlying function is continuous everywhere.

The Real Complexity

NeRF's rendering algorithm is solved in the mathematical sense: the volume rendering integral has a known closed form, and optimizing the MLP by gradient descent is guaranteed to find a local minimum. There is no NP-hardness or undecidability — the difficulty is purely computational.

  • Training the original NeRF takes 1–2 days on a GPU for a single scene, because every pixel requires hundreds of network queries and gradient back-propagation through all of them.
  • Rendering a single frame took minutes — far from real time.
  • The MLP must be queried per-point per-ray per-pixel, so even inference is expensive.

Researchers attacked these bottlenecks with structural insight: instead of evaluating an MLP everywhere, you can cache the learned field in a voxel grid (NeRF-based methods), a multi-resolution hash table (Instant-NGP by Müller et al., 2022 — 100× faster), or replace the implicit field entirely with explicit 3-D Gaussian splats (Gaussian Splatting, Kerbl et al., 2023 — real-time rendering).

The tradeoff is memory: an MLP is compact (a few MB), while hash grids and Gaussian splats can use hundreds of MB per scene. The search for the ideal accuracy / speed / memory triangle is still very active.

NeRF also struggles with unbounded scenes, dynamic objects, and specular reflections. Each limitation has spawned a sub-field: NeRF-W for in-the-wild lighting, D-NeRF for dynamic scenes, Ref-NeRF for reflections. Explore how neural network training generalizes across such architectures.

Where It Matters

The ability to reconstruct any viewpoint from a sparse set of photographs has immediate value across many fields:

  • Visual effects and film: NeRF-like representations let studios digitize actors and sets from video footage, enabling bullet-time effects and photorealistic digital doubles without expensive laser scanning.
  • Robotics and autonomous driving: a robot can build a NeRF of its environment from RGB cameras alone and query it for depth or occupancy without a dedicated LiDAR.
  • Cultural heritage: museums and archaeologists use NeRF to create high-fidelity digital twins of artifacts and sites, preserving them against damage or decay.
  • E-commerce: product pages can offer interactive 360° views generated from a handful of smartphone photos.
  • Medical imaging: NeRF-style volumetric reconstruction from sparse CT or MRI slices reduces radiation dose while preserving diagnostic quality.
  • Simultaneous Localization and Mapping (SLAM): NeRF is increasingly used as the map representation in real-time SLAM systems for AR/VR headsets.

NeRF's rise also accelerated interest in adjacent ideas — dimensionality reduction for the latent spaces that edit NeRF scenes, and neural network training strategies that make implicit scene representations converge faster.

Conclusion

NeRF distilled a 3-D scene into something as simple as a function call: give me a position and a direction, and I will tell you what you would see. By marching rays through that function and accumulating light with the volume rendering equation, it produces photorealistic images from viewpoints that no camera ever visited.

The original formulation was slow, but the insight — that geometry and appearance can be jointly encoded in the weights of a network and differentiated end-to-end — sparked an entire field. Five years later, descendants like Instant-NGP and Gaussian Splatting render NeRF-quality scenes in real time.

If you want to keep pulling the thread, the next concepts to reach for are the neural network training mechanics that make gradient descent converge on complex scenes, and dimensionality reduction techniques used to navigate the latent spaces where NeRF edits live.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/nerf/Content licensed under CC BY-NC 4.0.