Lensa ML
Lensa ML

Scaling Laws

Step 1 of 6: Power Laws

Straight lines on log-log axes

Exponent α0.076
Kaplan exponents:
0.010.11101M10M100M1B10B100B1T10× params×0.839Parameters N (log scale)L = 4.0 · (N / 1M)^−0.076illustrative: loss 4.0 at 1M params
Log-log slope
−0.076
Loss per 10× params
×0.839
Params to halve loss
9139×
Loss at 1T params
1.40

Each 10× in parameters multiplies the loss by 0.839, a 16.1% drop. Halving the loss takes a model 9139× larger, which is why language-model gains cost orders of magnitude.

Drag Exponent α, then click the three Kaplan exponents to compare model size, data and compute. Kaplan et al. (2020) measured these slopes on transformer language models and found the trends held over more than seven orders of magnitude, from tiny models to billions of parameters. Their form, L(N) = (N_c / N)^α, has no constant, so it predicts the loss falling toward zero forever. Later fits add an irreducible term E, the entropy of the text itself, which bends the line flat at very large scale.

Step 2 of 6: Training Compute

C ≈ 6ND floating-point operations

Parameters N6.74B
Training tokens D2.00T
1B10B100B1T10T10M100M1B10B100B1TC=10¹⁸C=10²⁰C=10²²C=10²⁴C=10²⁶8.1×10²² FLOPsParameters NTokens DC = 6 × 6.74B × 2.00T = 8.1×10²² FLOPs
Training compute
8.1×10²²
Tokens per param
297
A100-days
7501
Days on 1,000 GPUs
7.50

The 7B chat model on 2T tokens costs 8.1×10²² FLOPs: 7501 GPU-days at an illustrative 125 TFLOP/s per GPU, or 7.5 days on 1,000 GPUs.

Move Parameters N and Training tokens D, or click the 7B preset, and watch the point cross the dashed lines of equal cost. The factor 6 counts one multiply and one add per parameter per token in the forward pass, then about twice that in the backward pass. The rule ignores attention over the context, which is small when the model width is large next to the sequence length. The Llama 2 paper reports 184,320 A100 GPU-hours for its 7B model, about 7,700 GPU-days, close to this estimate.

Step 3 of 6: The Chinchilla Fit

Loss from parameters and tokens

Parameters N6.74B
Training tokens D2.00T
23451B10B100B1T10TE = 1.69, the irreducible lossfloor for 6.74B params: 1.875L = 2.023Training tokens D (log scale)Loss (nats/token)Loss = E + A/N^0.34 + B/D^0.281.69= 2.0205
Predicted loss
2.023
Irreducible E
1.69
Model term
0.185
Data term
0.148

The two terms are close (0.185 for size, 0.148 for data), so neither dominates. Loss is 2.023, which is 0.333 above the irreducible 1.69.

Set Parameters N, then sweep Training tokens D and watch the loss approach the floor for that size. Hoffmann et al. trained over 400 models, from 70M to more than 16B parameters on 5B to 500B tokens, and fitted this three-term formula to the final losses. The loss is in nats per token on their training text, so the numbers do not transfer to other datasets or tokenizers. Switch to 2024 refit to see a different set of constants fitted to the same data.

Step 4 of 6: Compute-Optimal Models

Best size for a fixed budget

Compute budget C (FLOPs)1.0×10²³
1.92.32.73.110M100M1B10B100B1T10T14.6B params, 1.14T tokensParameters N (tokens D = C / 6N fall as N grows)Loss
Optimal params
14.6B
Optimal tokens
1.14T
Tokens per param
78.2
Lowest loss
2.005

At 1.0×10²³ FLOPs the printed fit puts the optimum at 14.6B parameters and 78 tokens per parameter, and the ratio grows with C because α ≠ β. Chinchilla itself trained at 20; a 2024 refit of the same data lands much closer to it.

Drag Compute budget C and watch the bottom of the bowl move right as the budget grows. Setting the derivative of E + A/N^α + B/(C/6N)^β to zero gives N_opt ∝ C^(β/(α+β)) and D_opt ∝ C^(α/(α+β)). Hoffmann's two other methods both found exponents near 0.5, meaning parameters and tokens should grow equally. Chinchilla, at 70B parameters and 1.4T tokens, used the same compute as the 280B Gopher and beat it on most benchmarks. The printed constants disagree with those methods; Besiroglu et al. (2024) refit them and recovered a ratio near 20.

Step 5 of 6: Over-Training

Spend training compute to save on serving

Tokens for the 7B model2.00T
1.92.12.32.52.710²¹10²²10²³10²⁴compute-optimal frontier7B model, more tokens →297 tok/paramTraining compute C (FLOPs)Loss
Tokens per param
297
Extra loss vs optimal
0.007
Optimal model, same loss
12.4B
Extra compute, same loss
1.15×

At 297 tokens per parameter the 7B model spends 1.15× the compute a 12.4B compute-optimal model needs for the same loss. In return it is 1.84× smaller, so every generated token costs 1.84× fewer FLOPs to serve.

Drag Tokens for the 7B model from about 32B up to 20T and compare the rose path with the lime frontier. Generating one token costs about 2N FLOPs, so a model that will serve trillions of tokens pays for its size again and again after training ends. The original LLaMA paper made this argument for training small models far past the compute-optimal point. Llama 2 7B saw 2T tokens, about 300 per parameter, and Llama 3 8B saw about 15T, nearly 1,900 per parameter, with loss still falling.

Step 6 of 6: What Scaling Laws Miss

Repeated data, disagreements, sudden abilities

Epochs over the same data4
0U5U10U15U20U110203040if every token were newceiling 16.4UEpochs (U = unique tokens available)
Tokens seen
4U
Effective tokens
3.73U
Value of next epoch
82%

4 epochs give 3.73U effective tokens out of 4U seen, 93% of the value of fresh data. Chinchilla-style laws assume every token is new.

Click each tab: Repeated data, Kaplan vs Chinchilla and Sudden abilities, and move its slider. Muennighoff et al. (2023) found that up to about 4 epochs of repeated data is almost as good as fresh data; the curve uses their fitted constant R* ≈ 15.4. The Kaplan and Chinchilla rules differ partly because Kaplan's runs used one learning-rate schedule for every run length and smaller models. Loss laws also say nothing about data quality: filtered or synthetic text can shift the whole curve down.

What are scaling laws in machine learning?

Scaling laws are empirical formulas that predict a model's loss from its size, its training data and the compute spent on it. For language models the loss falls as a power law in each of these, so a plot on log-log axes is close to a straight line across many orders of magnitude. A law fitted on small, cheap runs can then forecast the loss of a run 1,000 times larger.

Labs use scaling laws to plan training runs before spending the compute. They decide how many parameters a model should have for a given GPU budget, how many tokens to collect, and whether a new architecture or data mix is worth scaling up. The 2022 Chinchilla paper used them to show that most large models of the time were too big for their data, which changed how open models such as Llama were trained.

Key ideas

  • Language-model loss falls as a power law in parameters, data and compute, which appears as a straight line on log-log axes.
  • Training a dense transformer costs about 6 FLOPs per parameter per token, so C ≈ 6ND.
  • The Chinchilla fit splits loss into an irreducible part, a term for model size and a term for data.
  • For a fixed compute budget, the best model grows roughly as the square root of compute, near 20 tokens per parameter.
  • Models meant for heavy serving are trained far past that point, because a smaller model is cheaper on every generated token.
  • Scaling laws predict loss, not specific abilities, and they assume fresh data of a fixed quality.

Common questions

How are scaling laws used to plan a training run?

A team trains a series of small models with the same recipe, fits a power law to their final losses, and extrapolates to the planned budget. The GPT-4 technical report describes predicting the final loss from models trained with 1,000 to 10,000 times less compute. The same fits help choose model size, token count and learning rate before the expensive run starts.

What is irreducible loss?

It is the part of the loss that no model size or amount of data removes, the E term in the Chinchilla formula. It reflects the randomness of text itself: many next tokens are not predictable from the context. Its value depends on the dataset and tokenizer, so an E of 1.69 nats per token applies only to the data it was fitted on.

Do scaling laws apply outside language models?

Yes. Similar power laws have been measured for image, video, speech and multimodal models, and for reinforcement learning. The exponents and constants differ by domain and architecture, so a law has to be refitted for each setting rather than copied from a language model.

Do scaling laws cover inference compute?

The classic laws cover training compute only. Newer work studies two other trade-offs: training smaller models longer to save inference cost, and spending more compute at answer time, through longer reasoning or many sampled answers, to raise accuracy on a fixed model. Both follow their own empirical curves.

Related explainers

References

  1. Kaplan et al. (2020), "Scaling Laws for Neural Language Models"
  2. Hoffmann et al. (2022), "Training Compute-Optimal Large Language Models"
  3. Besiroglu et al. (2024), "Chinchilla Scaling: A replication attempt"
  4. Muennighoff et al. (2023), "Scaling Data-Constrained Language Models"
  5. Schaeffer et al. (2023), "Are Emergent Abilities of Large Language Models a Mirage?"

Last updated