How to Read an AI Model Card: The Fields That Matter and Where They Come From

A model card is the document that describes an AI model: what it is for, how it was evaluated, and its limits. Every major vendor now publishes one, sometimes under another name - Anthropic and OpenAI also publish longer "system cards". The fields on this site's model pages are taken from these documents. This tutorial goes through what the fields mean.

Where model cards came from

The format was proposed in Model Cards for Model Reporting, first submitted on 5 October 2018 by Margaret Mitchell, Timnit Gebru and colleagues at Google. The paper argued that trained models should ship with documentation that gives "benchmarked evaluation in a variety of conditions", including across "different cultural, demographic, or phenotypic groups", along with the intended use and the details of how performance was measured.

It proposed nine sections:

1. Model details - who built it, when, version, type, licence.
2. Intended use - the uses it was built for, and uses out of scope.
3. Factors - groups, environments or conditions that could change performance.
4. Metrics - which measures were used and why.
5. Evaluation data - what it was tested on.
6. Training data - what it learned from, where that can be shared.
7. Quantitative analyses - the results, broken down by the factors above.
8. Ethical considerations.
9. Caveats and recommendations.

Vendor cards for today's large models do not all follow this layout, but most of these headings appear in some form.

Reading the specification fields

Context window. The maximum number of tokens the model can take in a single request - the prompt, any documents and the conversation so far. See tokens and context windows for what a token is.

Maximum output. A separate, usually much smaller limit on how many tokens the model can generate in one response. A model with a very large context window may still cap a single response far lower.

Knowledge cutoff. The date after which the training data does not reach. Vendors sometimes give two dates: the training data cutoff and a "reliable knowledge" cutoff, the point up to which they say the model's knowledge is dependable.

Modalities. Which inputs the model accepts (text, image, audio, video) and which it produces. These are often asymmetric - many models take images in but only return text.

Pricing. Usually quoted per million tokens, with input and output priced separately and output typically costing more. Many vendors add separate rates for cached input and batch processing. Comparison pages on this site show these side by side for two models.

Licence and weights. Whether the weights can be downloaded and under what terms. "Open weights" and "open source" are not the same thing: a model can publish weights under a licence that restricts commercial use. See running open-weight models locally.

Effort or thinking settings. Many recent models can be run at several reasoning levels. These are one model at different settings, not separate models, and a benchmark score is usually reported for a specific setting.

Reading the benchmark section

The single most useful question to ask of a benchmark table is: who ran it? Scores in a vendor's own card are the vendor's measurements, under conditions it chose - the number of attempts, the effort setting, the tools allowed. Independent leaderboards run models under their own fixed conditions, so the same benchmark can show different numbers in different places.

This site keeps the two apart. Vendor-reported figures are labelled as such in model guides, and independent results - from LMArena's human-preference voting, Epoch AI and others - are on the benchmarks page, each linked to its source.

Related

• How large language models work
• The NIST AI Risk Management Framework - which asks organisations to document and measure models in much the way model cards do.