NeRF Learns a Scene as a Function

NeRF stores a scene as a function you query. Posed photographs are enough supervision, because rendering is differentiable and the gradient reaches the field through the image.

Give the network a 3D location and a viewing direction. It returns density and color. Render enough rays through that function and you have a new view of the scene. The pipeline stores no mesh and no voxel grid.

The scene becomes a field you query instead of a table you store

The field maps coordinates to radiance and density. Density says whether something is there. Color says what light leaves that point in a given direction.

The representation stays continuous, so you can query between sampled points. A grid stores answers at fixed places, and a coarse grid caps the detail you can get back. A function computes an answer wherever you ask.

Rendering supplies the training signal

A ray leaves the camera and passes through the scene. The renderer samples points along the ray and asks the field for density and color at each one. It accumulates those samples into one predicted pixel. Training compares that pixel to the captured pixel at the same place.

The backward chain here is the one from the backprop note, extended through image formation. Every accumulation step is a differentiable operation, so the gradient walks back through the samples and into the field weights. Backpropagation updates the field until rendered views match observed views.

The loss rule from the linear classifier note holds without change. Posed photographs are the labels. Wrong means a rendered pixel that disagrees with the captured one. No 3D ground-truth mesh enters the loop.

View dependence is a feature of the representation

Color depends on viewing direction, so one point can look different to two cameras. That is how the field carries highlights and other view-dependent effects. The scene lives in the parameters of a function, and you reach it by querying along a ray.

The overview note listed the pressures a representation must survive, and pose sits first on that list. One field serves every camera, so view consistency becomes a property of the representation. The test is a camera pose the model never saw. Render it and look for geometry that drifts.

Positional encoding lets an MLP hold high-frequency detail

A plain MLP struggles to hold high-frequency detail when you feed it raw coordinates. Positional encoding maps each coordinate into a set of sinusoidal features. Those features give the network a handle on fine structure. The change happens before the first layer.

The architecture note called a design a bet about what structure the data has. That encoding places the bet at the input. The bet is that the scene holds fine structure the raw coordinates make hard to reach.

Classic NeRF pays for exactness on every ray

Each pixel costs many samples along its ray, and each image costs many rays. Per-scene optimization repeats that bill for every new scene. That makes classic NeRF slow to train and slow to render.

It also needs good camera poses and enough coverage. A pose that is off puts the ray in the wrong place. The field then fits the scene to that wrong ray.

Later work improved speed, generalization, and editing. The original design still earns study, because geometry and appearance become one differentiable training problem.

The Builder Test

Trace one pixel from the camera to the loss. Pick one pixel in one training image and name the ray it comes from. Write down the samples along that ray and the accumulation that turns them into a color. The captured pixel at the other end of that comparison is the label.

Now move the camera pose by a small amount and walk the same chain. The ray starts somewhere else, every sample moves with it, and the loss charges the field for the difference. You pass this test when you can point at the exact step where pose error becomes geometry error.

What Carries

A scene can be learned as a function when rendering gives the model a way to be wrong. Before you trust any new representation, find the operation that makes its error measurable. A representation shift changes what a model can express, and it changes nothing about how easily gradients turn against it.