A Prompt You Cannot Score Is a Preference

DSPy replaces prompt tweaking with a signature and a metric. The metric you pick becomes the system you get. Treat every model call as a module inside a program. Then optimize the program against a number you chose on purpose.

Real Tasks Need More Than One Call

One model call is convenient and narrow. A real task needs retrieval, decomposition, ranking, verification, tool calls, and retries, and none of that lives inside the call. A compound AI system is what you build around the model to cover the rest.

The compound system can beat the model alone because it adds structure. It decides what context to retrieve, how to ask, how to combine outputs, and how to examine them.

The overview note gave the alternative a name: demo thinking. A person supplies the missing state between steps, and the demo works because of the person. One giant prompt has the same shape, with the state buried in a paragraph nobody can score.

DSPy Turns Prompting Into Programming

DSPy reframes prompting as programming. A signature says what goes in and what must come out. Modules compose into a pipeline. An optimizer then searches for instructions, demonstrations, or parameters that move the metric you chose.

question -> retrieve passages -> draft answer -> verify against passages -> revise -> final answer with citations

Each arrow in that line is a place to measure. Most prompt work never gets that far. You change a sentence, look at the output, and hope it generalizes to the next question.

Define the task, define the metric, run the optimizer, then compare results. Comparison is the step that turns a change into evidence.

The Signature Is the Quiet Win

The signature is the part people skim. It forces you to name what information enters the module and what shape the module must return.

A signature turns "ask the model nicely" into a contract. With a contract you can test the module and swap it. You can also reason about how it composes with the modules on either side.

Ownership starts at the contract. When a run goes wrong, you want one question answered: which module owns the failure? Without named inputs and outputs, the answer is the whole pipeline, which is the same as no answer.

The Optimizer Gives You Exactly What You Measure

Optimization helps when the pipeline and the metric match the real job. Aim the optimizer at a cheap proxy and you get a system that is excellent at the proxy.

Metric overfitting is the standard failure. The optimizer moves whatever the score rewards, and the score climbs while the product gets worse.

• Reward answer length and the optimizer learns verbosity.
• Count citations on the surface and the system learns to cite without understanding.
• Leave latency and cost out of the score and the optimizer can build something impressive and unusable.

Bad tests produce bad software. Bad evals produce bad agents. The reasoning note drew the line: reflection happens inside the model, and verification happens outside it. A metric is that outside signal written down, and a weak metric is reflection with a number attached.

An evaluator decides whether a change improved the system or only made one demo prettier. Someone has to design that evaluator, so judgment moves earlier, into the metric and the module boundaries. That work happens before the first tuning run, and it is the expensive part.

Agents Have Programmable Insides

An agent looks like an open-ended loop from the outside. The inside is a set of modules you can program and score. Retrieval, tool choice, answer synthesis, and verification each take a named input and return a named output.

Teams add prompt steps until the examples look better. Without module-level evidence, the system gets slower and harder to debug while the real bottleneck stays untouched.

Attach examples and a metric at each interface, and a regression gets a return address. Keep the examples that expose regressions, and run them against every tuned version. Then you can show which component changed when the score improved.

The move is to replace vibes with interfaces. A coding agent needs files, tests, commands, and patches. A workflow agent needs state, permissions, rollback, and logs.

The agent-loop note put the rule plainly: the weakest interface sets the ceiling. A compound system makes those interfaces visible enough to rank.

The Builder Test

Take a system you run this week. List its interfaces: the tool schema, the retrieval quality, the state format, the evaluator, and the permission boundary.

Rank them by what a bad version of each costs you. The lowest one sets your ceiling, and prompt work does not raise it. I start at the evaluator, because a broken evaluator hides every other problem on the list.

Then write down which module owned your last bad output. If nobody can name it, name the interfaces before you tune anything else. Read the tuned program afterward. A pipeline no one on the team can follow is a black box you now own.

What Carries

A metric is only as good as the oracle behind it. Code carries the strongest oracle in this unit, because a test suite answers without an opinion. Take your weakest module to a task where the oracle runs itself, and find out whether your metric survives contact.