Evaluation Is a Contract

A benchmark number carries no meaning apart from the choices that produced it. Two of those choices decide leaderboards without ever appearing on them. Contamination puts the test set inside the training data, so a high score measures memory. Length bias makes an automatic judge prefer the longer answer, even when the longer answer hallucinates more.

Both survive because the reported number hides the pipeline that made it. Rebuild the pipeline and the number becomes readable, or it stops being worth reading.

What you measure becomes the target

Labs are ranked on a handful of numbers. Teams then change the data mixture, the architecture, and the training run to move those numbers. The number stops describing the model and starts designing it.

Four audiences read the same score for four different reasons.

• A company picking a model for one workflow wants to know whether it fits that workflow.
• A researcher tracking progress wants to know whether a training change improved reasoning.
• A policymaker weighing benefits against risks wants to know whether the model is safe enough to deploy.
• A developer mid-run wants feedback fast enough to steer the next checkpoint.

One evaluation cannot answer all four. An evaluation built to satisfy everyone satisfies no one. Pick the question first, then build the setup that fits it. An accuracy number with no context misleads all four readers at once.

Every evaluation is a pipeline of four choices

Four stages sit between a question and a number, and each one adds an assumption that the number hides.

The inputs set the ceiling. Prompt source and coverage live here, along with whether the rare and hard cases appear at all. In multi-turn chat the later inputs depend on earlier outputs, so you cannot script the sequence in advance.

The call decides what you are testing. Zero-shot, few-shot, chain-of-thought, tool use, and retrieval each expose a different part of the same weights. This stage also fixes the unit of measurement. You test a bare language model, an agent with tools, or a full system with scaffolding and routing.

The scoring decides what counts as right. Exact match, F1, pass@k, BLEU, and ROUGE reward different behavior, and each assumes the reference answers are clean. Cost belongs in this stage. Two models with the same score can differ widely in price and latency, and deployment pays that difference.

The reading decides what the number licenses. A score of 91 percent is not a deployment decision until you know which items fail. Separate real generalization from overlap with the training data. Then name what you measured: a model, a system, or a change in method.

Adapting prompts to one model buys realism and costs comparability. A fixed script pushes a chat model into states no user creates. Letting the model drive the conversation looks more like real use. In red-teaming, adapting each prompt to the last reply finds rare failures faster. Both moves make the inputs model-specific, so cross-model comparison stops being fair.

Every stage prices the next one. Evaluation is the stage that prices your reading of all the others.

Perplexity still guides pretraining and misses what users want

A language model assigns probabilities to token sequences. Perplexity measures how well it predicts tokens on held-out text, and lower is better. Pretraining optimizes this objective directly. Perplexity is also the metric the scaling notes fit, so those curves ride on this one number.

Earlier work compared n-gram models against neural models on Penn Treebank and WikiText. Large neural models cut perplexity by a wide margin, and that drop drove scaling.

GPT-2 trained on WebText and scored well out of distribution on those older benchmarks. That is transfer. GPT-3 and the models after it still report perplexity, and attention moved to downstream tasks.

Two mechanical properties keep perplexity in the loop. Token probabilities give smooth, fine-grained feedback that a discrete accuracy score cannot give. The same machinery reaches downstream tasks: condition on the prompt and score the candidate answers.

Perplexity is a blunt instrument for capability. A small change in behavior often leaves the number flat, so it judges small model changes poorly. A perplexity leaderboard also asks you to trust reported probabilities, and invalid or manipulated probabilities produce improvements that never happened. Name the proxy before you trust the number.

The open argument is whether matching the true data distribution is enough on its own. One side treats it as the whole problem. The other side calls that route inefficient and badly aligned with the tasks people run.

Multiple-choice benchmarks that compare probabilities over completions are perplexity under another name. LAMBADA asks for a missing word that needs long-range context. HellaSwag asks for the most plausible continuation as a proxy for common sense. Many draw on web-adjacent sources like WikiHow, so their items sit inside the pretraining corpus already.

Exams saturate, leak, and encode their own bias

MMLU covers 57 subjects with multiple-choice questions pulled from the web. It weighs knowledge heavily, and reasoning without the right facts does not score. Early evaluations used few-shot prompting, before instruction tuning existed. Prompt wording and example order both moved the scores. Instruction-tuned models now score well zero-shot, and parts of the benchmark are saturated.

MMLU-Pro raises the difficulty. It drops noisy questions and moves from four choices to ten. Strong models lose accuracy on it, and chain-of-thought helps on some items and not on others.

GPQA holds PhD-level technical questions that experts write and review. Experts score about 60 percent. Non-experts working with a web search score about 30 percent.

Google-proof means a plain search does not close that gap. Earlier models scored in the high 30s, and newer models score higher. A model can still reach the answer through web-like knowledge patterns.

Humanity's Last Exam is multimodal and hard by design, with prizes for items that defeat models. The people writing those items know how models behave, so the questions drift toward ultra-specific traps. That is a stress test at the far end. No user has a Tuesday that looks like it.

Every exam encodes the taste of the people who wrote it. Saturation, prompt sensitivity, and item bias all move a score that reads as pure capability.

Judges reward length

Open-ended instructions have many acceptable answers and no single reference output. Every automatic score for them is a trade-off written as a number.

Chatbot Arena shows a user two hidden replies and asks which one is better. Pairwise votes turn into ELO-style rankings. The prompts are live, no fixed test set exists to leak, and the human judgment is easy to give. The same design carries user bias, vote manipulation, and the leaderboard distortions that follow.

IFEval checks narrow constraints, like a word limit or a banned token. Those constraints are mechanical to check, so results reproduce across runs. They measure surface compliance, and a nonsense answer that obeys the word limit passes.

AlpacaEval hands the comparison to GPT-4 and reports win rates. Early versions preferred longer answers, and length-corrected variants cut that bias. Its scores correlate with Chatbot Arena. WildBench judges real human-bot conversations with LLMs working from checklists.

Length bias is the clearest case of a judge scoring the wrong variable. It lived inside a widely used benchmark until someone measured it and corrected for it. Assume your next judge carries a bias of the same kind, and find it before you publish.

Agent benchmarks score the model with its scaffolding: multiple calls, planning, tools, code execution, and debugging. The best systems still solve a small fraction of these tasks, and the fraction is growing. Long-horizon planning, tool use, and debugging expose weaknesses that short answers hide. Run one long task before you trust a short-answer score.

A model that refuses everything scores perfectly on safety

Safety benchmarks measure refusal, so a model that refuses everything maximizes the metric and serves nobody. Read safety numbers next to capability numbers, or do not read either.

HarmBench defines harmful behaviors and checks whether models refuse them. AIR-Bench maps safety categories onto laws and policies and probes models against those categories. Both aim at the same target: helpful work done safely, dangerous requests refused.

Two quantities hide under the word safety. Capability asks whether the model can produce harmful content at all. Propensity asks whether it does so under allowed prompts. Closed APIs lower propensity through safety tuning, and open weights expose capability. A user can strip the safety layer or fine-tune away its effect.

Jailbreak results are evidence about brittleness. Optimized gibberish suffixes push models past their safety rules, which puts refusal behavior on a thin layer. Systematic prompting surfaces harmful behavior behind nominal refusals.

Safety institutes run voluntary pre-deployment tests with labs on unreleased models, mostly reusing existing benchmarks. One question stays open: what counts as safe enough. Answer it in writing for your own product before a benchmark answers it for you.

Contamination and label noise break a benchmark without changing how it looks

Models train on corpora the size of the web, and benchmarks live on the web. Overlap is the default state, and the resulting score looks exactly like a clean one.

A filter on shared long n-grams catches the obvious cases. It misses translations, paraphrases, and variants, including the same math problem written in another language. Decontamination and deduplication are one operation seen from two ends of the pipeline. The data notes that follow run that filter across the training corpus. Run the same filter against your test sets and you have the decontamination check.

Many papers report no decontamination check at all. Publish the check beside the score, or state that you skipped it.

Label noise runs the other way. Benchmarks carry wrong answers and broken questions, and correcting those errors often lifts the top scores. Part of the difficulty a leaderboard reports is its own noise.

One more gap separates the tests from the work. Real traffic holds chit-chat, messy questions, spam, adversarial requests, and practical tasks. Benchmarks mostly quiz, where the writer knows the answer and uses it to grade. Users mostly ask, because they do not know the answer and need help. Most of the value sits in asking, and most of the measurement sits in quizzing.

Two projects show what closing that gap costs. Anthropic clustered real conversations and found coding to be a major use case. MedHELM asked clinicians which tasks matter in practice, then built benchmarks for those tasks. Both routes need real user or patient data, and that data carries privacy limits on collection and sharing.

The Builder Test

Write down what the evaluation cannot see, then decide how far to trust it. One paragraph covers it: which prompts it never contains, and which failure mode it does not score. Name the part of the score that the scaffolding earned.

Then attach that paragraph to the score every time you report it. A number that travels without its blind spot is a claim nobody can audit, including you.

What Carries

An evaluation earns trust when you can explain the choices that produced the number. That explanation is the contract, and the number is worth no more than the contract behind it.

Every benchmark score rests on what the model read before it met the test. The next note opens the diet.