How to Compare AI Models: Elo, Benchmarks, Price and Speed

In short: to compare AI models, look at a few separate measures side by side: how people rate their answers (LMArena Elo), how they score on fixed tests (benchmarks), what they cost per million tokens, how much text they can handle (context window), how fast they reply, and what they can take in and produce. No single number answers "which is better" - each measure tells you one specific thing, and this guide explains what each one means.

Why there is no single "best AI model" score

Different measures test different things, so a model can lead on one and trail on another. A model that people prefer for writing may be slower or more expensive; one that scores highest on maths tests may not be the one people pick in open-ended chat. Comparing well means knowing what each number measures and where it came from.

The measures below are the ones this site tracks for every model, each linked to its source.

1. LMArena Elo: which answers people prefer

LMArena's Elo rating comes from people comparing two anonymous answers and voting for the better one. On LMArena, a user asks a question, gets replies from two models without knowing which is which, and picks the one they prefer. Millions of these votes are turned into a rating for each model, a method described in the LMArena team's paper (Chiang et al., 2024).

How to read it:

• Higher is preferred more often. The gap matters more than the number. Under the Elo formula, a 100-point lead means the higher-rated model is expected to win about 64% of head-to-head votes; a 10-point lead is about 51% - close to a coin flip.
• Check the confidence interval. Each rating comes with a range. If two models' ranges overlap, the data does not clearly separate them, which is why LMArena can show several models sharing a rank.
• Note the effort setting. Many models can "think" harder before answering, and LMArena often lists the same model at different effort levels - for example "(High)" or "(Max)". These are one model at different settings, scored separately.
• It measures preference, not correctness. Votes reward answers people like, which usually tracks quality but can also reward style and length.

Our benchmarks table shows the current text leaderboard, with each model's effort level and rank.

2. Benchmarks: scores on fixed tests

A benchmark is a fixed set of questions or tasks with known answers, so every model can be scored the same way. Common examples:

• GPQA Diamond - hard graduate-level science questions written by experts (Rein et al., 2023).
• SWE-bench - real GitHub issues the model has to fix in real code (Jimenez et al., 2023).
• FrontierMath - very hard, unpublished maths problems.

How to read them:

• Ask who ran the test. A score in a company's own announcement was measured by that company, under conditions it chose - number of attempts, effort setting, tools allowed. Independent groups such as Epoch AI run models under one fixed setup, so their numbers are directly comparable with each other but may differ from the vendor's.
• Compare the same benchmark and version. "SWE-bench" and "SWE-bench Verified" are different tests; so are different versions of an index.
• A benchmark covers one skill. A high maths score says little about writing or coding.

On our model pages, vendor-reported scores are labelled as such, and independent scores link to Epoch AI.

3. Price: what it costs per million tokens

API prices are quoted per million tokens, with input (what you send) and output (what the model writes) priced separately. A token is a chunk of text, about three-quarters of an English word - see what a token is.

To compare cost fairly:

• Look at both prices. Output usually costs several times more than input, so a model that writes long answers costs more.
• Work out a real request. Multiply your typical input and output by each price. For example, at $2 input and $10 output per million, a 10,000-token prompt with a 1,000-token answer costs $0.02 + $0.01 = $0.03.
• Count "thinking" tokens. Reasoning models are often billed for the tokens they spend thinking, which can make an answer cost more than its visible length suggests.
• Check discounts and conditions. Cached input, batch processing, introductory prices and long-prompt surcharges can change the bill a lot.

4. Context window: how much it can read at once

The context window is how many tokens a model can consider in one request - your prompt, documents, conversation history and its reply. A 1-million-token window holds roughly 750,000 words. Check the maximum output too: many models can read far more than they can write in one reply.

5. Speed: how fast it answers

Two numbers describe speed: output speed (tokens per second once it starts writing) and time to first token (how long before the first word appears). On this site both come from OpenRouter's live measurements of real traffic. Speed varies by provider - the same model can run faster on one host than another - which is why model pages also show the fastest single provider.

6. What it can do: inputs, outputs, tools and licence

Before comparing numbers, check that a model can actually do the task:

• Inputs and outputs - text only, or also images, audio, video or PDFs; text replies only, or also images or speech.
• Tools - web search, code execution, function calling.
• Availability - in an app, through an API, or only for some customers.
• Licence - closed and API-only, or open weights you can download and run yourself.

Example: three models side by side

	Gemini 4 Argon	Claude Opus 5.5	GPT-6.1 Sol
LMArena text	1524 Elo (rank 1, high effort)	1508 Elo (rank 3, high effort)	1484 Elo (rank 22, max effort)
Input / output price (per 1M tokens)	$2 / $10	$4 / $20	$2 / $10
Context window	1.05M	1M	1.05M
Output speed (OpenRouter median)	39 tokens per second	147 tokens per second	61 tokens per second

These figures update automatically from our model database. Build your own comparison of any two models on the compare page, or see every model in the full rankings table.

A simple way to compare models

1. Write down what you need - the kind of task, how long your inputs are, and whether you need images, audio or tools.
2. Filter by capability - drop models that cannot take your inputs or fit your context length.
3. Compare quality - LMArena Elo for general preference, and the specific benchmark closest to your task.
4. Compare cost on a real request - using your own input and output sizes.
5. Check speed - if people will be waiting for the answer.
6. Test with your own prompts - leaderboards are averages; your task may differ.

Related

• How LLMs work: a beginner's guide
• How to read an AI model card
• AI model rankings and benchmarks
• Compare two AI models