Transfer Learning Is Buying a Head Start

Two numbers decide the transfer plan: the size of your labeled set, and its distance from the pretraining data. Everything else follows from those two, including where you freeze. The common failure is reaching for a pretrained backbone by reflex, without asking whether the borrowed features match the job.

Training a large vision model from scratch is usually the wrong default

Random weights hold no visual features, so scratch training pays for edges and textures you can borrow. Start from a model that already learned those features on a large dataset. You then have two moves: use it as a feature extractor, or fine-tune part of it for your task.

Both moves rest on the same bet, which is that the source task learned features your task can reuse. Early visual filters, textures, shapes, and object parts transfer even when the final labels change. You replace the classifier head because it was trained to name a different set of classes.

Data size and domain similarity give you four cases

Similarity here means how close your images are to what the pretrained model already saw. Small and similar is the easy corner. A frozen backbone and a new head are enough. A large similar dataset earns more room, so fine-tune deeper and trust the result more.

The dangerous corner is small and different. The borrowed features can be wrong for your images, and you do not have enough data to learn better ones. Large and different makes full fine-tuning plausible, and pretraining on your own domain becomes a real option.

Feature extraction is fast and has a ceiling

The pretrained network turns your images into embeddings, and a small classifier trains on top. This is fast and stable, and it overfits a small dataset less easily than a full fine-tune. The ceiling arrives when the frozen features miss something specific to your domain. The head cannot invent what the backbone never encoded.

A frozen backbone bets on coverage. The pressures the source images spanned, pose, lighting, scale, background, occlusion, must also cover yours. When they do not, the head reads a representation that already discarded the signal you need.

Fine-tuning must adapt the features it inherited

Fine-tuning updates part or all of the pretrained network. The features you want are already in there, so every update is a disturbance you must keep small. Use a smaller learning rate for the pretrained layers and a larger one for the new head. A reckless learning rate erases the head start you paid for.

Unfreeze from the top down. Early layers hold generic edges and textures. Later layers hold dataset-specific semantics, which is where your task differs most. That order is a default, and error analysis overrides it.

Fast improvement is exactly when leakage hides

Transfer makes performance rise fast, and a fast rise covers a dirty split. Check that near-duplicate images do not cross train and validation before you trust the validation curve. The split is still the only instrument you have, and a borrowed backbone does not repair a broken one.

Check your preprocessing against what the pretrained model expects. That match is the same train-test path agreement the note on setup failures demanded. You now inherit that contract from a model you did not train.

One more leak sits upstream. The pretraining set can already contain your evaluation images, and no split of your own data will show it.

The Builder Test

Freeze the backbone and train the head. Record the errors on the hard slices. Pull the confident mistakes first, which is the same inspection the visualization note used to catch a shortcut. Unfreeze deeper blocks only when that analysis shows the representation itself is the limit.

What Carries

Transfer is a bet that the source features are reusable here, and the error slices are how you settle it. Transfer moves a representation between tasks. The next problem needs a representation that moves through time.