Few-Shot Prompting and In-Context Learning: What Examples in a Prompt Do

Few-shot prompting means putting a handful of worked examples in the prompt so the model can infer the task from them. The model's weights do not change; it adapts to the examples within a single request. That behaviour is called in-context learning. For general prompt-writing practice, see the Prompt Engineering Guide.

Where the terms come from

The vocabulary was set by the GPT-3 paper, Language Models are Few-Shot Learners, submitted on 28 May 2020 by Tom B. Brown and 30 co-authors at OpenAI. It evaluated a 175-billion-parameter model "without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model."

The paper defined three settings:

• Zero-shot - a description of the task only, such as "Translate English to French:".
• One-shot - the description plus one example.
• Few-shot - the description plus several examples. In the paper this was typically 10 to 100, as many as fitted in the model's 2,048-token context window.

A few-shot prompt for sentiment labelling looks like this:

Review: The battery lasts two days. -> positive
Review: It stopped charging after a week. -> negative
Review: Setup took five minutes. -> positive
Review: The screen scratches easily. ->

The paper reported that performance generally improved from zero-shot to few-shot, and that the gap grew with model size.

What in the examples actually matters

Later research tested which parts of the examples models use.

Formatting and the label space matter more than correct labels. Rethinking the Role of Demonstrations (Sewon Min and colleagues, 2022) replaced the correct labels in few-shot examples with random ones and found performance dropped only slightly across a range of classification and multiple-choice tasks. What mattered more was that the examples showed the set of possible labels, the kind of input text to expect, and the overall format.

Order and balance introduce bias. Calibrate Before Use (Tony Z. Zhao and colleagues, 2021) found GPT-3's few-shot accuracy could swing widely depending on which examples were chosen and in what order. It identified three biases:

• Majority label bias - favouring whichever label appears most often in the examples.
• Recency bias - favouring the label of the last example or examples.
• Common token bias - favouring answers that are frequent in the model's training data.

The paper's fix, contextual calibration, measures the model's bias on a content-free input such as "N/A" and adjusts the output probabilities to cancel it.

Few-shot today

Context windows have grown far beyond GPT-3's 2,048 tokens - the models page lists each model's current window - so far more examples fit in a prompt than in 2020. Vendors' own prompting documentation continues to describe examples as a way to show a model the output format wanted. The findings above still describe what is being tested when examples change: their format, the labels they cover, and their order.

Few-shot examples that include written reasoning, rather than answers alone, are the basis of chain-of-thought prompting.