Scaling Laws Make Expensive Choices Less Blind

Test loss falls on a straight line on log-log axes. That line is predictive enough to pick model size and token count before you spend the budget.

A frontier run is expensive, so you make the choices before it starts. Train many small models and fit the curve. The fit picks architecture, hyperparameters, data mixture, and the split between parameters and tokens. Cheap measurement buys the expensive decision.

Loss falls on a straight line on log-log axes

The line has a formula with three fitted numbers: a floor, a coefficient, and an exponent.

Here n is the dataset size and alpha is positive. L infinity is the error that more data does not remove. Drop that term from the fit and the extrapolation promises loss the large run never reaches. The same shape holds when you plot against parameters or against total compute.

Across many domains and models the loss stays monotone and log-log linear over wide ranges. Inside those ranges the measurements show three regions. Tiny datasets sit near random, a wide middle band follows the power law, and the curve flattens at the floor. Fit inside the middle band, because the two ends bend any fit that includes them. Hestness and colleagues reported that shape in 2017 for translation, speech, and vision.

Fitting a curve and extrapolating it predates this field by decades. A Bell Labs line of work in 1993 argued that full training on huge datasets costs too much. It told you to train smaller systems, fit an irreducible error plus a decaying term, and extrapolate. Banko and Brill showed smooth gains from more data in early NLP. That picture still frames how people plan runs today.

Classical learning theory answered the same question with worst-case bounds from VC dimension and Rademacher complexity. Those bounds run loose. Modern practice measures the loss curves of real networks and fits the formula directly.

The slope reads as difficulty

Strip the floor and the reducible part carries the slope:

On log-log axes that is a straight line with slope minus alpha. For a baseline, estimate the mean of a Gaussian with variance sigma squared from n samples. The squared error falls like one over n, which is a slope of minus 1.

The reported slopes for deep networks are far shallower.

• About minus 0.095 for language modeling
• About minus 0.13 for machine translation
• About minus 0.3 for speech

Each one is slower than minus 1 and slower than minus 0.5. Neural networks behave like non-parametric estimators in a high-dimensional space. An estimator that bins a D-dimensional region and averages locally gives rates near n to the minus one over D. A large D flattens the slope.

Read the slope as difficulty. A shallower slope means a higher effective dimension, and language modeling is the shallowest of the three. A unit of loss in language modeling costs more data than the same unit in speech.

A better data mixture moves the offset and leaves the slope alone

Change the data mixture and the whole curve moves up or down. The slope moves only a little. The offset tells you how good the mixture is at every size.

So compare mixtures with small runs and take the better offset. The advantage continues on the runs you cannot afford to repeat.

When fresh data runs out, repeat what you have. Repetition helps, and each epoch returns less than the one before it. After a few epochs the gains drop sharply.

Price the repeats by replacing the raw token count with an effective sample size. That effective size grows more slowly as the repetition grows. The second pass over a corpus is worth less than the first.

At a trillion tokens the choice gets concrete. You repeat the clean sources, or you add noisier new ones. Each source mixture carries its own curve, so fit both and let the curves set the repetition and the growth.

Small runs decide architecture, optimizer, and shape

Kaplan-style comparisons train several architectures across compute levels. The transformer line sits below the LSTM line by an almost constant factor. For the same loss, an LSTM pays much more compute.

Efficiency decides how much scale you can afford. The tokenizer note made that case for the input, and this curve makes it for the architecture.

Google ran architecture sweeps that compared many variants against a transformer baseline. A few variants beat it, including GLU and mixture-of-experts. Most of the rest added overhead and left the scaling where it was.

Optimizers separate the same way. Across sizes the Adam curve sits below the SGD curve, and the gap holds as an almost constant factor.

Very shallow transformers do poorly. Past a small depth threshold, a broad band of depth-to-width ratios performs the same, with matching slopes and small offsets. Tune the aspect ratio on small models and trust it on the large one.

Batch size and learning rate carry their own curves, and the second scaling note measures both at production sizes.

Chinchilla lands at about twenty tokens for each parameter

A fixed compute budget forces a split between parameters and tokens. A tiny model cannot use a huge corpus. Reverse it and you waste the capacity, because the model stays undertrained.

The joint fit gives each axis its own decaying term:

Here n counts tokens and N counts parameters. These surfaces fit real measurements and predict a large run from small ones.

The compute side comes from the accounting note. A training run costs about six times parameters times tokens in FLOPs, written here with D for the token count.

Both optima grow near the square root of compute, so the two stay proportional to each other. Double the budget and you buy a bigger model and more tokens in the same breath.

Chinchilla measured that split for dense language models with three methods.

• Lower envelope: plot loss against compute for many runs. Read N and n at the best point of each budget, then fit how both scale with FLOPs.
• IsoFLOP sweeps: for each budget, sweep the model size and adjust the tokens to hold the FLOPs fixed. The loss against N is U-shaped, so fit its minimum.
• Surface fitting: fit the joint surface and solve for the compute-optimal point.

The first two agreed. The third carried a bug in the fit from the original paper. Correct that fit and all three agree. The rule stands: for compute-optimal dense training, use about twenty tokens for each parameter.

Training-optimal and product-optimal are different budgets

Chinchilla minimizes the training loss for a fixed training compute budget. That answer is training-optimal. You pay training compute once. You pay inference on every request.

Products pay for inference, and that cost grows with the parameter count and with the context length. When inference dominates, you buy a smaller model trained on more tokens for each parameter. You accept the higher training cost.

The published ratios bracket the choice. GPT-3 sits near two tokens for each parameter, undertrained by the Chinchilla rule. Chinchilla sits near twenty. Newer systems train past twenty to cut the serving cost for each unit of quality.

The Builder Test

Name the decision the curve changes. If it moves no model size, no token count, no compute budget, and no risk, the plot is decoration.

Control the runs behind the curve, with one axis moving and the rest held fixed.

The real choice stays the same at every budget. You are spending fixed compute across parameters, tokens, and quality. The curve makes that split arguable in advance.

What Carries

A scaling law is a planning tool. It holds while the measured assumptions hold, and those assumptions are the part you must write down.

The constants move when data quality moves, when the optimizer changes, when the architecture changes, or when the evaluation leaks. Extrapolating past the setup that produced the curve is where the plan breaks.

The fit is cleanest on cross-entropy and held-out perplexity from the training distribution. Downstream benchmarks behave less smoothly. A model that scales well in perplexity can still be weak on capabilities.

Scaling describes training-like regimes well. Far outside that regime the behavior turns irregular, and on adversarial or out-of-distribution tasks it can reverse. The Inverse Scaling Prize collected tasks where larger models do worse. The task often punishes a behavior that scaling strengthens, such as copying.

The rule optimizes training compute and stays silent on what serving costs. The KV cache note prices that bill.