Neural Radiance Fields (NeRF)

Imagine handing a computer thirty photographs of a coffee mug — taken from different angles — and asking it to show you the mug from a viewpoint that no camera ever captured. Before 2020 that was a research problem spanning decades. Then NeRF arrived and made it almost trivial.

Neural Radiance Fields (Mildenhall et al., 2020) represent a scene not as a mesh or a point cloud, but as a continuous function: given any 3-D position $(x, y, z)$ and a viewing direction $(\theta, \phi)$, a small neural network outputs the color $(r, g, b)$ and volume density $\sigma$ at that point. The network has no explicit geometry — it simply learns to interpolate what the scene looks like from any direction.

To actually produce an image, NeRF fires a ray through each pixel and marches along it, sampling the network at many points and accumulating color using the volume rendering equation — the same physics that describes light passing through fog or smoke. The whole system is trained end-to-end by minimizing the difference between rendered and real photographs.

The result is a compact, differentiable 3-D model that can be queried for any novel view, with no manual segmentation or 3-D scanning required.

The heart of NeRF is volume rendering: a ray travels through space, and at each sample point the scene's neural network votes on color and opacity. Step through the demo to see how samples accumulate into a final pixel color.

<!-- {{c_html_comment}} -->
<div class="controls">
  <label>{{label_scene}}: <select id="sceneSelect">
    <option value="sphere">{{scene_sphere}}</option>
    <option value="box">{{scene_box}}</option>
    <option value="fog">{{scene_fog}}</option>
  </select></label>
</div>
<canvas id="rayCanvas" width="360" height="220" title="{{canvas_title}}"></canvas>
<div class="step-row">
  <button id="prevBtn" type="button" disabled>&#8592; {{btn_prev}}</button>
  <span id="stepLabel" class="step-label">{{lbl_step}} 1 / 8</span>
  <button id="nextBtn" type="button">{{btn_next}} &#8594;</button>
</div>
<div id="sampleInfo" class="sample-info">{{info_init}}</div>
<div class="accum-row">
  <span class="accum-label">{{lbl_pixel}}</span>
  <canvas id="pixelCanvas" width="36" height="36" title="{{pixel_title}}"></canvas>
  <span id="accumText" class="accum-text">R=0 G=0 B=0 T=1.00</span>
</div>
<button id="resetBtn" type="button" class="ghost">{{btn_reset}}</button>

/* {{c_css_comment}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
canvas { display: block; border-radius: 8px; border: 1px solid #cdd9e3; background: #0a0e14; }
.controls { margin-bottom: .5rem; font-size: .9rem; }
.controls select { font-size: .85rem; padding: .2rem .4rem; border-radius: 4px; border: 1px solid #adb1b8; }
.step-row { display: flex; align-items: center; gap: .6rem; margin: .5rem 0; }
.step-label { font-size: .85rem; color: #555; flex: 1; text-align: center; }
button { font: 600 13px system-ui; padding: .4rem .8rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button:disabled { opacity: .4; cursor: default; }
button.ghost { background: #fff; color: #1d3557; }
.sample-info { font-size: .82rem; color: #333; min-height: 2.8em; padding: .3rem .1rem; line-height: 1.5; }
.accum-row { display: flex; align-items: center; gap: .6rem; margin: .3rem 0; }
.accum-label { font-size: .82rem; color: #555; }
#pixelCanvas { border-radius: 4px; border: 1px solid #adb1b8; }
.accum-text { font: 600 .8rem ui-monospace, monospace; color: #333; }

// Code not found

Notice how early opaque samples dominate — once a ray hits dense material, later samples contribute almost nothing. That is why NeRF can represent sharp object boundaries even though the underlying function is continuous everywhere.

NeRF's rendering algorithm is solved in the mathematical sense: the volume rendering integral has a known closed form, and optimizing the MLP by gradient descent is guaranteed to find a local minimum. There is no NP-hardness or undecidability — the difficulty is purely computational.

• Training the original NeRF takes 1–2 days on a GPU for a single scene, because every pixel requires hundreds of network queries and gradient back-propagation through all of them.
• Rendering a single frame took minutes — far from real time.
• The MLP must be queried per-point per-ray per-pixel, so even inference is expensive.

Researchers attacked these bottlenecks with structural insight: instead of evaluating an MLP everywhere, you can cache the learned field in a voxel grid (NeRF-based methods), a multi-resolution hash table (Instant-NGP by Müller et al., 2022 — 100× faster), or replace the implicit field entirely with explicit 3-D Gaussian splats (Gaussian Splatting, Kerbl et al., 2023 — real-time rendering).

The tradeoff is memory: an MLP is compact (a few MB), while hash grids and Gaussian splats can use hundreds of MB per scene. The search for the ideal accuracy / speed / memory triangle is still very active.

NeRF also struggles with unbounded scenes, dynamic objects, and specular reflections. Each limitation has spawned a sub-field: NeRF-W for in-the-wild lighting, D-NeRF for dynamic scenes, Ref-NeRF for reflections. Explore how neural network training generalizes across such architectures.

The ability to reconstruct any viewpoint from a sparse set of photographs has immediate value across many fields:

• Visual effects and film: NeRF-like representations let studios digitize actors and sets from video footage, enabling bullet-time effects and photorealistic digital doubles without expensive laser scanning.
• Robotics and autonomous driving: a robot can build a NeRF of its environment from RGB cameras alone and query it for depth or occupancy without a dedicated LiDAR.
• Cultural heritage: museums and archaeologists use NeRF to create high-fidelity digital twins of artifacts and sites, preserving them against damage or decay.
• E-commerce: product pages can offer interactive 360° views generated from a handful of smartphone photos.
• Medical imaging: NeRF-style volumetric reconstruction from sparse CT or MRI slices reduces radiation dose while preserving diagnostic quality.
• Simultaneous Localization and Mapping (SLAM): NeRF is increasingly used as the map representation in real-time SLAM systems for AR/VR headsets.

NeRF's rise also accelerated interest in adjacent ideas — dimensionality reduction for the latent spaces that edit NeRF scenes, and neural network training strategies that make implicit scene representations converge faster.

NeRF distilled a 3-D scene into something as simple as a function call: give me a position and a direction, and I will tell you what you would see. By marching rays through that function and accumulating light with the volume rendering equation, it produces photorealistic images from viewpoints that no camera ever visited.

The original formulation was slow, but the insight — that geometry and appearance can be jointly encoded in the weights of a network and differentiated end-to-end — sparked an entire field. Five years later, descendants like Instant-NGP and Gaussian Splatting render NeRF-quality scenes in real time.

If you want to keep pulling the thread, the next concepts to reach for are the neural network training mechanics that make gradient descent converge on complex scenes, and dimensionality reduction techniques used to navigate the latent spaces where NeRF edits live.