Adversarial Examples Expose the Gap Between Scores and Sight

The gradients that trained the model are the gradients that break it. A defense means nothing until you name the attacker.

A tiny perturbation, invisible or irrelevant to a human, moves a network into a confident wrong prediction. The overview opened this unit with a pressure list: pose, lighting, scale, background, occlusion, class imbalance. An adversary is that list with intent. It searches for the one shift you never tested.

Differentiability makes the attack cheap

A neural network is differentiable from the loss back to the pixels. Training walks that path to the weights. The attack walks the same path to the input. It nudges each pixel toward a higher target score, or a lower score on the correct class.

The visualization note computed that same input gradient and drew it as a saliency map. Saliency asks which pixels the model leans on. The attack asks which pixels to move. One backward pass answers both.

A high softmax score is confidence under the learned representation

A high softmax score reports confidence under the learned representation. It carries no claim about whether the image is unambiguous, safe, or aligned with human sight. A perturbation can push the model across a decision boundary while a human sees the same object.

That gap decides how much weight the score deserves. It matters most in systems that touch money, safety, identity, health, or physical control.

Clean held-out accuracy is the number the classification note warned about, and the adversary attaches an optimizer to it. Your held-out set arrives by sampling. The adversarial set arrives by search.

FGSM takes one step, PGD takes many

The Fast Gradient Sign Method takes one step in the direction that increases loss with respect to the input. PGD takes many smaller steps and projects each one back inside the perturbation budget. It is usually the stronger first-order attack.

One step assumes the loss surface stays straight across the whole budget. Many small steps drop that assumption and follow the curve, which is the whole difference between the two. A defense tested only against FGSM reports a number PGD does not support.

A defense without a named attacker is a wish

A robustness claim only means something inside a threat model. Name what the attacker can change: pixels within an L-infinity bound, a physical sticker, lighting, camera angle, or compression. Say what the attacker can examine, including your weights, your preprocessing, and the defense itself. The success condition decides the rest, because any wrong label and one specific wrong label are different attacks.

Change any one of those three and the defense changes with it. A method that holds under an L-infinity budget can fail completely against a sticker on the object. Write the threat model before you write the defense.

A defense that hides gradients looks strong until someone adapts

Gradient masking, image compression, and naive preprocessing all lower attack success on the first try. They damage the gradient the attacker needs. The attack then fails for a reason that has nothing to do with robustness. A weak evaluation reads that failure as a defense.

An adaptive attack that knows the defense reads it correctly. It usually shows the model was only hard to attack with the first method you tried. The honest standard is harsh: evaluate against attacks that know your defense.

The options that survive that standard are adversarial training, certified robustness in limited settings, better evaluation protocols, and system-level safeguards. Each one costs training time or narrows the setting where the claim holds.

The Builder Test

Before you ship a vision model, run four evaluations and report them in one place.

• Clean held-out accuracy.
• Accuracy under corruptions.
• Accuracy under distribution shift.
• Accuracy under an adaptive attack that knows your defense.

A defense that moves only the first number has moved nothing. After launch, keep watching real failures. The threat model you did not write is the one production finds.

What Carries

Accuracy says how often the model is right on the dataset you chose. Robustness asks how badly it fails when the world stops cooperating. The reliability work that catches most of your real failures is older and duller than any attack. Do that work first, then earn the right to worry about the attacker.