Capabilities Are Interfaces

A capability belongs to the model and the interface together. The same checkpoint looks weak under a bare prompt and strong under a benchmark-friendly one. The claim stays incomplete until it names the prompt, the context, the decoding rule, the benchmark, and the failure examples.

A Capability Appears Through An Interface

Language models handle language modeling, question answering, translation, arithmetic, article generation, and grammar correction. They also handle tasks nobody trained them for. Which of those you see depends on how you ask.

Capability claims disagree because the setups disagree. One evaluation sends a bare prompt. Another gives examples, tool access, retrieval, scratch space, and repeated attempts. The checkpoint stays the same across all of them. The interface decides what it can express.

One sweep settles which pair a claim describes. Compare bare prompting, few-shot prompting, tool access, retrieval, scratch space, and repeated attempts on one task. A fragile model scores well under one of those and drops under the rest. If the score swings across that sweep, the claim belongs to the model-interface pair rather than to the checkpoint.

Few-Shot Prompting Writes A Temporary Task Specification

The GPT-3 era made this visible. Engineers scaled a next-token model and few-shot behavior came out of it. Place a few examples in the context window, ask for a continuation, and the model often infers the task format.

Show three examples of country to capital and the model continues the pattern. Phrase the same game badly and it misses. The examples tell the model which game you play. The checkpoint holds no module for that game.

Those examples write a temporary task specification inside the context window. Change the examples and you change the task. Pattern completion behaves like task learning here, and the behavior leaves when the prompt does.

A Benchmark Is A Contract

Benchmarks such as LAMBADA, HellaSwag, TriviaQA, WebQuestions, and arithmetic probes slice capability into observable tasks. Each one declares what counts as input, output, success, and failure. That declaration is the contract the benchmark offers, and it covers only what it names.

Language modeling benchmarks measure predictive fit, usually with perplexity or likelihood. Question answering measures whether the model connects prompt context with stored or retrieved knowledge. Translation needs mapping between languages. Arithmetic needs symbolic precision at every step. A perplexity win and a question answering win are two different claims about one checkpoint.

Scoring moves the number as much as the prompt does. An exact match, a judged answer, and a human rating measure three different things. Record which of the three produced the number you quote. Then check whether the benchmark scores the interface you ship. If it scores a different one, the number belongs to a different product.

Record The Whole Interaction Contract

The benchmark contract covers the task. The interaction contract covers everything around it. The common product mistake is to compare models on one task with no fixed interface. That comparison ranks prompts and publishes the ranking as a model verdict.

The interaction contract has eight fields: prompt format, examples, context, decoding, tools, retries, refusal policy, and scoring. User expectation belongs beside them, because the same output passes one reader and fails another. Write all of it next to the number. Skip it and the number measures an undocumented system.

Decoding belongs in the record even when you never change it. The introduction note set up the decoding fork, and that fork sits inside every capability number you report. Change the decoding rule and the score moves while the weights stay fixed.

The Average Hides The Slice That Fails

Treat novel task behavior as evidence to investigate. The evidence that settles it lives in the failures, so pull the wrong answers and read them. Name what they share, then check whether the shared property appears in the work you plan to ship.

Emergent behavior means a capability became visible under a certain scale and interface. Stability is the open question after that. A model that answers the benchmark and breaks in a messy user workflow gives you a number with no coverage.

Say a model can do a task without saying under what conditions and the measurement turns into a myth. The conditions are cheap to write down. The myth is expensive to unwind after a launch.

The Builder Test

Write the interaction contract for one task you own, every field, in one file. Hand the file to another engineer and ask for the number. If the number they get does not match yours, a field is missing from the file. A contract nobody else can run is a note to yourself.

What Carries

A capability is never separate from the way you ask for it and the way you score it. An interface elicits it, and that interface must travel with the claim. Name the interface and a surprising result turns into an engineering decision.

You can now say what a model does under a named interface. The next question is who pays when that interface fails.