Scaling Laws: From Kaplan (2020) to Chinchilla (2022)

Scaling laws are measured relationships between the resources put into training a language model and how good the result is. Two papers set the terms of the discussion: one from OpenAI in 2020, and a revision from DeepMind in 2022 that changed how large models were trained.

Kaplan et al. (2020): loss follows power laws

Scaling Laws for Neural Language Models was submitted on 23 January 2020 by Jared Kaplan and nine co-authors at OpenAI. It trained many Transformer language models and measured their test loss - the cross-entropy that measures how well a model predicts the next token, where lower is better.

Its central finding: "The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude."

Two further findings shaped what followed:

• Shape matters little. "Other architectural details such as network width or depth have minimal effects within a wide range." Total parameter count mattered far more than how the parameters were arranged.
• Bigger models are more sample-efficient. Larger models reached a given loss with fewer training tokens. The paper concluded that, for a fixed compute budget, the efficient choice was a very large model trained on a relatively modest amount of data, stopped well before convergence.

That conclusion matched the direction of the field at the time, toward ever-larger parameter counts.

Chinchilla (2022): scale data as fast as parameters

Training Compute-Optimal Large Language Models was submitted on 29 March 2022 by Jordan Hoffmann and 21 co-authors at DeepMind. It asked the same question - given a fixed compute budget, how large should the model be and how much data should it see? - and tested it by training "over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens."

The answer differed from Kaplan's. The authors found "the model size and the number of training tokens should be scaled equally": "for every doubling of model size the number of training tokens should also be doubled." By that measure, the large models of the day were significantly undertrained - too many parameters for the data they had seen.

The test: Chinchilla against Gopher

To check the prediction, DeepMind trained Chinchilla with the same compute budget as its earlier model Gopher, but split differently:

	Gopher	Chinchilla
Parameters	280B	70B
Training tokens	300B	1.4T

Chinchilla, a quarter of Gopher's size and trained on about four times the data, outperformed Gopher, GPT-3 (175B), Jurassic-1 (178B) and Megatron-Turing NLG (530B) across a large range of evaluations. The paper reported an average accuracy of 67.5% on the MMLU benchmark, more than 7 percentage points above Gopher.

Being smaller also made Chinchilla cheaper to run after training - a point the authors raised, since inference cost scales with model size.

Why the two papers disagreed

The Chinchilla paper attributes much of the difference to method: among other things, Kaplan's runs used a fixed learning-rate schedule regardless of training length, which made shorter training runs look less effective than they were. Chinchilla's fits used schedules matched to each run's length.

What followed

Chinchilla's ratio works out to about 20 training tokens per parameter (1.4 trillion / 70 billion). Many later open-weight models were trained on far more tokens per parameter than that. Chinchilla minimised training compute for a given quality; training a smaller model for longer costs more compute up front but produces a model that is cheaper to serve. This is part of why vendors now publish smaller models alongside their largest ones, as on the models page, and why parameter count alone says little about how a model ranks on the benchmarks page.

Related

• Attention Is All You Need - the architecture these laws were measured on.
• Foundational AI Research Papers Worth Reading