Preference Optimization Is Proxy Design

Push a proxy reward hard enough and the true human win rate turns down. The proxy score keeps climbing the whole time. That curve is why the reasoning systems that work now optimize rewards a program can check.

Preference optimization makes two decisions, and both belong to you. You decide what counts as good. You decide how hard to push on it.

Push the proxy and the true win rate turns down

The setup is pairwise preference data. A human sees two model responses to a prompt and picks the better one. A reward model learns from those picks, and the policy maximizes that reward. The reward model is the proxy, and the rest is a question of how far to trust it.

DPO removes the separate reward model. It writes the reward as a log ratio between the target policy and a reference policy. That ratio goes into a Bradley-Terry preference model, and training becomes supervised. The update raises the likelihood of the preferred response more than the dispreferred one. It grows when the implied reward disagrees with the human pick.

Raise the optimization pressure on that proxy and two curves separate. The proxy reward climbs without limit. The true human win rate climbs, turns, and falls. In that gap the policy is fitting the reward model, and the preference behind it has stopped moving.

Preference labels are noisy. Human preferences are complex and incomplete, so the reward model learns a partial picture. The optimizer finds the part of that picture that is wrong and lives there.

Students ran RLHF with three reward sources. They used real human preferences, noisy AI feedback, and clean noiseless AI feedback. The curve appeared for human preferences and for noisy AI feedback. Clean noiseless rewards almost erased it. The noise produces the turn.

I have stopped reading a rising reward curve as progress. It reports the proxy, and the proxy is the thing under attack.

Pretraining and supervised fine-tuning fit likelihoods, so the model keeps a probabilistic view of the world. RLHF optimizes a reward. In experiments the RLHF models became overconfident, most visibly at temperature 1. They scored worse on calibration than the SFT baselines.

Calibration is absent from the reward, so nothing in the objective protects it. Stop treating an RLHF model as a well-calibrated probability estimator. If a downstream system reads probabilities out of the model, keep an SFT checkpoint for that job.

The preference stage is also worth less than the first papers suggested. AI2 measured PPO ahead of DPO. Later work, Tulu 3 included, did not reproduce that ordering. A better supervised stage gave most of the gain either method gave. In one experiment SFT alone matched RLHF, and only length-normalized DPO added a small gain.

Results move with the base model, the data, and the environment. One paper does not settle the ordering.

Verifiable rewards remove the noise that caused the failure

Human approval is noisy, easy to game, expensive, and prone to overoptimization. The classic RL successes ran on rewards that were stable, cheap, and scalable. Keep the RL machinery and change the task.

Math, coding, logic, and puzzles have clear correct answers or strong automatic judges. A program decides the reward, and the label noise that bent the curve goes away.

The scaling-law note bought an expensive decision with cheap measurements. A verifier runs the same trade for reward. It costs an answer key or a unit test, and it returns the same verdict every time.

This change is the base of the o1-style models. DeepSeek R1, Kimi K1.5, and Qwen 3 all sit on it.

A verifier removes the noise inside the proxy and leaves the proxy standing. The distance between a correct answer and a good answer stays open. Clean verifiers cover a small part of what people ask a model to do. Every one of those pipelines still ends in RLHF for the rest.

PPO works and costs a value model the size of the policy

Policy gradient maximizes expected reward under the policy. The gradient raises the probability of the high-reward samples. It lowers the low-reward ones.

The naive version is fully on-policy. Every update needs fresh rollouts and fresh reward computations. For a language model the rollout is the expensive part. You want several optimization steps out of each batch of rollouts.

TRPO reuses samples from an older policy through importance sampling. Likelihood ratios correct the samples. A KL constraint holds the new policy near the old one and blocks very large steps.

PPO replaces that constraint with clipping. It multiplies the likelihood ratio by an advantage term. Then it clips the ratio to a band such as [1 - ε, 1 + ε]. The clip removes any gain outside the band, so one update cannot move the policy far.

PPO also carries a value network that predicts expected returns. Generalized advantage estimation lowers the variance of that estimate. Two more parameters, γ and λ, set the balance between bias and variance.

For a language model the setting is a contextual bandit. The prompt is the input. The action is one full generated sequence. One scalar reward arrives at the end, and the task has no real state transitions.

Training still runs at token level. The task reward lands at the end and reads as correct or incorrect. The loop broadcasts that final scalar backward across the tokens. Per-token KL penalties hold the policy near a reference policy.

The outer loop is ordinary. Collect rollouts, compute rewards and advantages, take a clipped policy step, then clip the gradients. The cost sits in the parts list.

• a reward model
• a value model about as large as the policy
• a GAE implementation with its own tuning
• KL penalties for each token or for each sequence
• many small tricks that decide whether the run stays stable

The run then holds the policy, a reference copy for the KL penalty, a reward model, and a value model. The value model alone doubles the memory and makes the training harder.

PPO is clean on the page. Every version that survives contact with a large model carries that parts list, and GRPO exists to cut it.

GRPO drops the value model and uses the group as the baseline

GRPO keeps the PPO-style clipped objective and removes the two most expensive parts, the value model and GAE. For each prompt you sample a group of G responses. Each response gets a scalar reward.

The group mean and the group standard deviation turn that reward into an advantage.

That advantage goes into the clipped objective, and no value network appears anywhere in the loop.

The group is every response to the same question. The group mean is then a baseline that carries the difficulty of that question. Subtract it and the variance drops. Learning depends on relative success inside one prompt, and the standard deviation normalizes the scale across prompts.

The baseline arrives for free in parameters and gets paid for in rollouts. You sample G responses for every prompt, so the cost moves out of a second network and into generation.

GRPO can add clipping or KL terms to hold the policy near a reference. In a fully online single-step configuration you can drop the clipping. You are left with a policy gradient and a baseline.

DeepSeekMath is where GRPO earned the default position on math. It ran two reward types. An outcome reward pays 1 for a correct final answer and 0 otherwise. A process reward model scores the reasoning steps instead.

GRPO with outcome rewards beat fine-tuning on correct outputs. It also beat the online variant that refreshes those outputs from the current model.

The original GRPO objective pushes chains of thought longer for the wrong reason

You can subtract any baseline that does not depend on the action. The policy gradient stays unbiased and the variance usually drops. Division is a different operation, and the theorem does not cover it.

Dividing by the group standard deviation changes the gradient. The update no longer matches the true policy gradient, and the damage shows when that standard deviation is small.

The standard deviation goes small when a group agrees with itself. Every answer is correct on an easy problem. Every answer is wrong on a hard one. Division inflates those advantages, and training drifts toward both extremes. The mid-difficulty problems carry the learning signal, and those are the ones the update walks away from.

The original objective also divides the total reward by the output length. On a wrong answer the reward sits near zero and the KL penalties are negative. A longer sequence spreads that negative number across more tokens and lowers the per-token penalty. The model learns to fail at length.

On correct answers the same normalization pushes toward shorter outputs. A model that is stuck takes the other branch and runs a long pathological chain of thought.

Dr-GRPO removes the division by the standard deviation and removes the length normalization from the reward. Accuracy held or improved on tasks such as GSM8K. The uncontrolled length growth stopped, and the output length settled at a reasonable plateau.

That plateau is the evidence. Some of the very long chains in GRPO runs are artifacts of the objective. High performance does not need them.

R1-zero is where the length story started. It begins from DeepSeek V3 before instruction tuning and before RLHF, with no chain-of-thought SFT. It runs RL with verifiable rewards, mostly on math. A binary accuracy reward scores the answer, and a format reward keeps the reasoning inside think tags.

The format reward is doing quiet work. It keeps the reasoning contained and machine-readable, which is what makes the rest of the pipeline possible.

As RL continues the chains get longer. The paper reads that as learning to think longer on harder problems. It reports aha-style behavior such as backtracking. After Dr-GRPO that reading needs a discount.

A length-biased objective produces some of the same growth. Large base models produce aha-style text from SFT alone. R1-zero still shows the real result. Outcome-based RL on verifiable math rewards turns a base model into a strong reasoner.

If your GRPO run grows chains without an accuracy gain, audit the objective before you credit the model.

The evaluation note named length bias as one of two things that decide a leaderboard without appearing on it. The post-training note showed every shortcut in the feedback taught back as behavior. Here the same bias arrives through the objective, which is the third door into one failure.

R1, Kimi, and Qwen converge on one recipe

Three systems from the same window landed on the same four stages. Long chain-of-thought SFT comes first. Then RL on verifiable outcomes, then general RLHF, then distillation into smaller models.

R1 aims at production use across tasks. It starts from DeepSeek V3 and fine-tunes on many long reasoning traces. The source of those traces is not stated, and distillation from strong models is the likely answer. The goal is a model that produces long structured reasoning easily and stays readable through RL.

Even a small amount of chain-of-thought data raises math scores for base models such as Qwen 2.5.

The reasoning RL stage repeats R1-zero with two additions. Format rewards enforce the thinking tags. A language consistency reward stops the chain of thought from mixing languages at random.

After reasoning RL the model is strong on math and logic and less friendly. They run SFT on mixed reasoning and nonreasoning tasks. Then they run GRPO-like RL on preference data, which obeys the DeepSeek V3 pipeline.

Distillation closes the pipeline. R1 generates traces and answers, and smaller models train on them. Those distilled models improve on math benchmarks against base models of the same size.

Kimi K1.5 arrived at about the same time and matches or beats o1 on many reasoning benchmarks. Its edge comes from the data work. They tag math questions by domain and balance the domains. They drop multiple-choice and true/false items, which are easy to guess or game. They keep questions a program can check, which means short text answers, numbers, and code outputs.

Difficulty selection does the rest. They run the base SFT model on each problem without reasoning and sample several answers. They keep the problems that stay hard, for example the ones that fail best-of-8. RL then spends its budget where the model is weak.

A chain-of-thought SFT step comes first here too. The data and prompt details are thin in the paper. The goal is a warm start that already produces usable reasoning traces.

Their objective starts from expected reward with KL regularization to the base policy. A nonparametric trick like the one in DPO writes the implied rewards as policy ratios. They then penalize the squared difference between the implied rewards and the actual rewards. The gradient looks like a policy gradient with a batch-mean baseline and an explicit squared KL term.

Qwen 3 runs the same pipeline and adds discipline to the data. They decontaminate against the validation benchmarks and filter the SFT data by hand. That pass removes low-quality chains and guessing-style chains. Their reasoning RL uses about 3,995 examples and still produces strong gains. On a strong base model, RL on verifiable tasks is data-efficient.

Chain-of-thought length is now a design parameter

The inference note priced generation in bytes moved, since every sequence drags its own KV cache. A long chain of thought is that bill in a new form. Kimi treats length as a control variable and pays for it directly.

RL on language models is system-heavy for the same reason. Rollout generation and training updates run at once, and rollouts are slow. Weights move from the RL workers to the inference workers, and trajectories move back. Long chains widen the length variance and make batching worse.

In each batch they find the shortest output and the longest output, then place every output on that range. A correct answer gets a length reward that pulls toward the short end. A wrong answer gets pulled toward the middle, away from both very short and very long. Correct solutions compress, and long wrong solutions earn nothing.

Length pressure applied early collapses the model into short wrong answers. The schedule has two phases. The model first learns to solve the problem. The stronger length reward arrives after that and compresses the reasoning while accuracy holds. With length control the average chain grows early, then flattens.

Qwen 3 moves the control to inference. One model carries two modes. The think mode produces a chain of thought before the answer, and the no_think mode answers directly. Special tokens in the prompt select the mode, and the SFT data shows both behaviors.

Forcing a stop during training teaches the model to switch from thinking to answering. At inference you set a budget on thinking tokens and force the answer when the budget runs out. Test-time scaling is smooth in both directions. More thinking tokens raise accuracy, and fewer lower it slowly with no cliff.

Set that budget per request and price it like any other serving cost.

The stages pull against each other. Reasoning RL raises math and STEM scores. Thinking mode fusion keeps those gains and adds the control. General RLHF improves instruction following and general tasks. It costs a small amount of thinking-mode math and STEM performance.

That is the price of a friendly generalist next to a reasoning specialist. You pick which one you are shipping. Data curation, reward design, and inference cost now matter as much as the choice of RL algorithm.

The Builder Test

Run the reward past the point where the score still improves. Then measure the thing you wanted, on held-out cases the reward never touched. If the score keeps rising while that measure falls, you have found the turn. Finding it on purpose costs one run.

Before you trust an improvement, name the proxy out loud. Say what the reward paid for and what it ignored. Then find the cheapest way to raise that score without doing the work. The optimizer will find that path too.

Supervised examples teach format. Preference methods and verifier signals teach which answers to make more likely. A model learns the shape of a helpful answer, the length a reward prefers, and the refusal pattern. Reliability does not follow.

What Carries

Reward design is model design. A reward a program can check is a reward you can audit after the run. A reward that only a human panel can judge drifts, and the optimizer finds the drift first.

Alignment spans data design, objective design, evaluation, the refusal and usefulness trade-off, monitoring, and rollback. One fine-tuning run is one item on that list.

Post-training teaches the model what the system rewards. Whether that matches what you meant is a separate measurement, and you must run it.

PPO, GRPO, and Dr-GRPO all rest on one estimator of the policy gradient. The next note takes that estimator apart.