Optimization Is How Models Pay for Being Wrong

Optimization is repeated local correction. The loss curve is the instrument that names which correction is broken. The height you descend is the loss chosen in the note on linear classifiers. The coordinates are the parameters. Once a score function and a loss exist, training becomes a movement problem.

The job is to move downhill without wasting steps and without walking into nonsense.

The gradient is local advice about a surface you cannot see

A gradient says how the loss changes when you nudge each parameter. It is local advice about the point you stand on. It names the direction that rises fastest right here, and descent steps the other way.

In a real network the parameter space has millions or billions of dimensions. You do not visualize that surface. You probe it, one local measurement of slope at a time.

The gradient never points at the best answer. It points at the steepest local change. It earns its keep when you repeat small corrections in the direction it gives.

Random search is the embarrassment baseline

One crude method samples random weights and keeps the best. It ignores the shape of the loss surface and treats learning as lottery tickets. It must make you uncomfortable.

Run it once anyway, as the floor. It gives the first loss value to beat, which makes the next run comparable. An optimizer that cannot beat random poking is reporting a broken setup. Look at the data path and the loss.

SGD trades an exact gradient for many more updates

Computing the gradient over the full dataset at every step costs too much. Stochastic gradient descent uses minibatches instead, and the estimate it returns is noisy. A full-dataset step is exact and rare. A minibatch step is noisy and frequent, and frequency wins the wall clock.

The noise buys something. It helps the optimizer escape sharp local quirks. It also makes the step size matter more, because a full stride along a rough direction overshoots.

The learning rate decides whether descent is careful, slow, or chaotic

The learning rate is the size of the step you take blindfolded. Too small and training crawls. Too large and the loss jumps or explodes. A stable run moves downhill at a pace the surface supports.

Sweep the learning rate before you change anything else. A run that crawls and a run that explodes both name that knob first.

Read the curve before you touch the architecture

The mistake is to treat gradient descent like a button. Optimization stays empirical even when the math is clean. Your instruments are the loss curve, the gradient norms, the validation behavior, and the update magnitudes.

• The loss is flat from the first step. Examine the learning rate, the gradients, and whether the data reaches the model at all.
• The loss explodes. Examine the step size, the initialization, and the loss for numerical safety.
• Training improves and validation stalls. The model is learning the training set more than the task, so examine overfit and leakage.
• Both improve slowly. The setup can work, and the step size or the capacity is underpowered.

A gradient norm at zero and a loss that will not move are one finding on two instruments. Learning rate, labels, batch size, initialization, and implementation bugs announce themselves here before final accuracy does.

Minibatch noise complicates the reading. The per-step loss is jagged by construction, so a single step is not a reading. Compare a window of steps against the window before it. A jagged curve trending down is doing its job. A jagged curve that stays flat sends you back to the list above.

Numerical gradients are slow truth

A numerical gradient changes one parameter by a tiny amount and watches the loss move. An analytic gradient computes the derivative directly, and training runs on it because it is fast. Numerical gradients are too slow for training. They are the reference that says whether the analytic ones are right.

When you write a new loss or a new layer, check it on a tiny input first. Numerical gradients are affordable there. The same habit has a larger version. Before a long run, make sure the model can overfit a tiny batch. Both checks are cheap, and both stop a broken setup before it costs you hours.

The Builder Test

Predict the shape of the loss curve before you launch the run. Write the prediction down, launch, and compare the first steps against it. Name the mismatch as one of four things: scale, learning rate, data, or gradients.

Sweeping the knob until one curve looks good teaches you nothing about what was broken. The prediction turns a run into evidence. A curve that matches confirms the setup. A curve that misses names the bug.

What Carries

A single gradient can be noisy, local, or badly scaled. The curve over many steps is the evidence.

A gradient helps only when it reaches every parameter. The next note is the bookkeeping that delivers it.