How to Run an LLM Locally

In short: An open-weight model runs on your own computer when its file fits in your computer's memory (graphics card memory plus normal RAM). Small models such as Gemma 4 E2B need about 3 GB and run on phones and old laptops. Mid-size models such as Gemma 4 31B, Qwen3.8 27B and gpt-oss-20b need about 16 to 20 GB. The largest open models, Kimi, DeepSeek, GLM and MiniMax, need 90 GB to over 600 GB even in their smallest compressed versions. Free apps such as Ollama and LM Studio download a model and open a chat with it; each has its own install guide, linked below.

What you need

A computer that meets the app's own requirements. These are the figures each developer publishes:

Free disk space for the model itself. Each model is a separate download of 1 GB to 20 GB for the models in the first table below, and far more for the largest ones. Ollama's documentation says models "can be tens to hundreds of GB in size".

Enough memory for the model. The model has to fit in graphics card memory plus RAM. If it fits only partly in the graphics card, the rest runs on the processor and generation is slower (the low-end hardware guide has measured examples). The conversation itself (the context window) takes extra memory on top of the model.

How these models rank on LMArena

LMArena ranks models by people's votes in blind side-by-side comparisons. The chart shows the score and overall rank of each model in this guide that LMArena ranks, read live from the benchmarks table. Models that are not on the benchmarks table, such as Gemma 4 E2B and E4B, are left out.

Open models that fit on a normal computer

The table lists models whose 4-bit version fits in 24 GB of memory or less. "Download" is the size Ollama shows for that tag on its model library, read on 1 November 2026. "Memory" is the figure the source in brackets publishes for the 4-bit version: Google's table for Gemma 4 counts the model weights plus 20% overhead; Unsloth, which publishes the compressed (GGUF) files most local apps use, counts graphics memory plus RAM. The LMArena rank is read live from the benchmarks table.

Gemma 4 E2B and E4B are made for phones and laptops: the "E" stands for "effective" parameters, and Google's model overview explains that their total memory is higher than the effective parameter count suggests. Gemma 4 26B A4B and Qwen3.5 35B A3B are mixture-of-experts models: only 4B and 3B parameters work on each token, which makes them fast, but Google's overview notes that "all 26 billion parameters must be loaded into memory".

How fast they run: measured speeds

Speed is shown as tokens per second (a token is roughly three quarters of a word). Vendors rarely publish speeds for ordinary hardware. The exception is Google, whose LiteRT team publishes measured speeds for Gemma 4 E2B and E4B on phones, laptops and a Raspberry Pi (E2B card, E4B card; 1,024 prompt tokens, 256 generated, 2,048 context):

For other models, the figures below come from people who ran them and published their setup. They used llama.cpp, the engine inside Ollama and LM Studio, unless noted:

Unsloth's gpt-oss guide gives a floor rather than a measurement: 14 GB of RAM, or graphics memory plus RAM, for "6+ tokens per second" with gpt-oss-20b.

The largest open models: Kimi, DeepSeek, GLM, MiniMax, Qwen

The open models highest on LMArena are far larger. Their weights are free to download, but even the smallest compressed versions need workstation or server memory. The figures are Unsloth's, from each model's run-locally guide, as total graphics memory plus RAM (or a Mac's unified memory):

Two techniques make some of these runnable on less: keeping the model's experts in normal RAM while the graphics card does the rest, and reading part of the model from a fast SSD. Both are slower. Measured results people have published:

Unsloth's guides run these models with llama.cpp, and the low-end hardware guide shows how llama.cpp splits a model between the graphics card and RAM.

Install an app to run them

Each app has its own step-by-step guide, written for people who have never opened a terminal:

The quickest start, on Windows: install Ollama from ollama.com/download/windows, open PowerShell, type ollama run gemma4:e2b and press Enter. It downloads Gemma 4 E2B (4.6 to 7.5 GB) and opens a chat.

Related

• How to run an LLM on low-end hardware: 4 GB to 8 GB graphics cards, processor-only PCs, Raspberry Pi and phones.
• Open source LLM leaderboard: every open-weight model in LMArena's order.
• Run open models locally: llama.cpp, Ollama and vLLM: how the engines differ, quantisation and the local API.
• Open-weight AI models explained: what the licences allow.