Chain-of-Thought Prompting: What the Research Actually Showed

Chain-of-thought (CoT) prompting means getting a language model to write out intermediate reasoning steps before its final answer. It is one of the few prompting techniques with a clear research origin and measured results. This article covers what those papers found. For the basics of writing prompts, start with the Prompt Engineering Guide.

The original paper: worked examples

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models was submitted on 28 January 2022 by Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma and colleagues at Google.

The method is a variation on few-shot prompting. Instead of showing the model example questions paired only with answers, each example also shows the reasoning:

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.

The headline result: prompting a 540-billion-parameter model (Google's PaLM) with just eight such exemplars reached state-of-the-art accuracy on GSM8K, a benchmark of grade-school maths word problems - surpassing, the authors report, a fine-tuned GPT-3 with a verifier.

The paper also found the benefit depended on scale. Chain-of-thought helped large models and did little for small ones, whose written reasoning was often fluent but wrong. The authors described it as an emergent ability of model scale.

Zero-shot: one sentence instead of examples

Four months later, Large Language Models are Zero-Shot Reasoners (Takeshi Kojima and colleagues, submitted 24 May 2022) showed the worked examples were not always needed. Adding the phrase "Let's think step by step" before the answer was enough to make the model reason in steps.

With InstructGPT (text-davinci-002), the paper reports accuracy rising:

Benchmark	Standard zero-shot	With "Let's think step by step"
MultiArith	17.7%	78.7%
GSM8K	10.4%	40.7%

Self-consistency: sample several chains

Self-Consistency Improves Chain of Thought Reasoning in Language Models (Xuezhi Wang and colleagues, 2022) added a second step. Instead of taking one reasoning chain, sample several with some randomness, then take the answer that appears most often - a majority vote over reasoning paths. The idea is that a complex problem usually has several valid routes to the correct answer, while mistakes scatter across different wrong answers.

How built-in reasoning changed the picture

The papers above prompted models that answered immediately unless told otherwise. Many current models reason before answering as part of how they were trained, and expose that as a setting rather than a prompt. The model pages on this site list these as effort or thinking settings - for example, a single model offered at low, medium, high and max effort, where higher settings spend more reasoning tokens before the answer. With such models, the amount of step-by-step reasoning is controlled by that parameter, and vendors document how their own models respond to prompts asking for it.

What carries over from the research is the underlying finding: on multi-step problems, generating intermediate steps before the final answer measurably changed accuracy, and the size of the effect depended on the model.

Related

• Few-shot prompting and in-context learning
• Benchmarks and rankings - including the maths and reasoning benchmarks used to measure this.