Name the Job Before You Name the Family

Generative modeling is a set of contracts, and the contract you name chooses the model family. Name the family first and you spend the project defending a choice you cannot test.

The job is to learn enough structure from the training distribution. With that structure you can assign probability, sample, reconstruct, interpolate, edit, and reason about missing data. The phrase make new images covers all of that and distinguishes none of it.

Each job in generative modeling asks for a different contract

Likelihood estimation, sampling, reconstruction, representation learning, controllable editing, compression, and missing-data reasoning are separate jobs. Each one names a different output and hands you a different kind of control over the image distribution. Treat them as one job and every comparison afterward is muddy.

A job that needs likelihood takes an explicit probabilistic model, one that scores data cleanly. Sharp visual samples sit closer to adversarial and diffusion-style objectives. Latent-variable methods are the ones to look at when the job is a compact representation.

Each family answers a different question

The generative-models note walked the mechanics, PixelRNN and PixelCNN, autoencoders, VAEs, and GAN training. Underneath those mechanics sit three different objectives.

An autoregressive model predicts the next visible piece from the pieces before it. A VAE compresses an image into a latent variable, decodes it back, and keeps that latent space sampleable. The GAN objective trains a generator until a discriminator cannot separate its samples from real data. Each objective carries its own strengths and its own failure modes.

A latent space is a bet that hidden factors explain the pixels

Latent-variable models bet that lower-dimensional hidden factors explain high-dimensional images. An organized latent space makes interpolation and sampling behave, and a tangled one reconstructs but does not generalize gracefully.

Reconstruction error looks the same in both cases. Walk a path between two latents and watch whether the decoded images stay plausible. When representation is the job, that walk is the evaluation.

Sampling makes the model show its distribution

A classifier hides behind one accuracy number. A generative model must put its distribution on the page. The samples show blur, mode collapse, missing global structure, texture shortcuts, and memorization. You can see the failure before you compute anything.

Name the metric, then name what the metric hides

Likelihood, sample sharpness, diversity, speed, controllability, reconstruction, and downstream features each judge a different promise. Sharpness does not report diversity, which is why the list carries both. Every metric on that list hides a failure, and a second measurement has to catch it.

The classification note warned about the clean accuracy number, and a clean likelihood number buys no more trust. A shallow pass compares samples by taste. A serious pass compares promises by evidence.

The Builder Test

Before you name a family, write three lines. The output the system needs goes on the first. The metric that judges it goes on the second, and the failure that metric hides goes on the third. If the third line stays empty, the comparison is not ready to run.

A family chosen because its samples looked good in a talk fails the first line.

What Carries

The overview note opened on one question, which is what representation survives an image nobody arranged for the model. Samples are that representation made visible, and generation answers the question with pictures rather than a score. Name what your representation must survive: pose, lighting, scale, background, occlusion. Then choose the machine that survives it.