Visualizing ConvNets Is Debugging the Representation

A visualization earns its place only when it changes a decision. Collect different data, fix a shortcut, reject a deployment, add a test. A picture that moves none of them is decoration.

The useful job is narrow. A probe catches the model using the wrong evidence while you can still act on it.

Feature-space neighbors show what the model thinks is similar

Pull the nearest neighbors of an image in raw pixel space and you retrieve similar color and similar background. Pull them in the feature space of a trained ConvNet and the neighbors share meaning instead. That difference is the finding. The network built a space where visual meaning is easier to compare.

Run the same retrieval across the training set and it pays a second time. Tight clusters that mix two labels mark unclear class boundaries. Exact repeats mark duplicates. An image sitting deep inside the wrong cluster is a candidate for a bad label. The case worth an hour is the neighbor that looks wrong to you and correct to the model.

A t-SNE map generates questions and proves nothing

t-SNE compresses high-dimensional features into a two-dimensional map, and similar images often cluster there. Use it for intuition and for error inspection. The clustering is not evidence of structure, because compressing to two dimensions discards what it cannot fit. A cluster that surprises you buys one thing, which is a question you can settle with examples.

The valuable visualization is the one that catches the shortcut

A bird classifier can learn sky. A medical model can learn scanner artifacts, and a product classifier can learn background color. All three stay accurate on the split you measured. They keep that accuracy until the deployment distribution changes, and users meet the failure before you do.

The classification note sorted misclassified images by confidence before trusting any number. The shortcut hunt uses the same sort. Pull the confident mistakes first, because a confident mistake is where a shortcut hides.

Saliency is a probe, and every probe distorts

Saliency asks which pixels most affect the class score. Deconvolution-style methods and gradient visualizations try to show what a neuron or a layer responds to. Run either one over a ConvNet and you read the local shared filters the convolution note built.

The builder question stays concrete. The model uses the object, or it uses a shortcut: background, watermark, color cast, or border artifact.

Deconvnets, gradients, feature inversions, and t-SNE each distort the truth in their own way. None of them replaces a quantitative test or error analysis. The requirement is the same for all four. A probe earns its run only when it hands you a testable next question. That question points at a shortcut, a class confusion, or a feature dependence you can verify.

Confidence is confidence under the learned representation

A ConvNet can be confidently wrong on an image that looks meaningless to a human. The same model can flip on a perturbation a human barely notices. The representation it learned is not human perception.

A ConvNet stays useful under that limit. A confidence score is a statement about the model, and it holds only inside the representation the model learned. State the limit before anyone acts on the number.

The Builder Test

Take the class your model confuses most often. Pull feature-space neighbors for its confident mistakes and look at what those neighbors share. Name that shortcut in one sentence. Then write the quantitative test that falsifies it, and run the test before you touch the model.

What Carries

A probe leaves behind two things, a picture and a test. Keep the test and let the screenshot go. Once you can see what a representation learned, the practical question is whether you can borrow it.