Running Open-Weight Models Locally: llama.cpp, Ollama and vLLM

An open-weight model is one whose trained weights can be downloaded and run on your own hardware, rather than reached only through a vendor's API. The models page shows each model's licence and whether it can be self-hosted. Running one needs inference software. Three open-source projects cover most of that ground, and they are built for different situations. Everything below comes from each project's own documentation.

llama.cpp

llama.cpp describes itself as a "plain C/C++ implementation without any dependencies", with the goal to "enable LLM (and VLM) inference with minimal setup and state-of-the-art performance" across a wide range of hardware.

• Hardware. The README calls Apple silicon "a first-class citizen", optimised through ARM NEON, Accelerate and Metal. It ships custom CUDA kernels for NVIDIA GPUs, and lists Vulkan, SYCL, HIP for AMD and MUSA for Moore Threads GPUs.
• Quantisation. It supports 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit and 8-bit integer quantisation. Storing weights in fewer bits shrinks memory use, which is what lets larger models fit on consumer hardware, at some cost in output quality that grows as the bit count falls.
• File format. Models are converted to the GGUF format; the repository includes a convert_hf_to_gguf.py script for converting models published on Hugging Face.
• Serving. It includes llama-server, which the README describes as an "OpenAI-compatible API server".

Ollama

Ollama packages local model running into a single application. Its README describes it as a way to "get up and running" with open models including Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen and Gemma.

• Platforms. macOS, Windows and Linux, plus an official ollama/ollama Docker image.
• Use. One command downloads a model and opens a chat - the README's example is ollama run gemma4.
• API. A REST API on localhost port 11434, with a chat endpoint at /api/chat.
• Licence. MIT.

vLLM

vLLM is aimed at serving models to many users at once rather than one person on one machine. Its documentation says it was "originally developed in the Sky Computing Lab" at UC Berkeley and is now maintained by a wide community.

• PagedAttention. Its core technique manages the attention key-value cache in fixed-size blocks, borrowing the idea of paging from operating systems, so memory is not reserved for the longest possible response. The method was published in Efficient Memory Management for Large Language Model Serving with PagedAttention.
• Throughput features. Continuous batching of incoming requests, chunked prefill and prefix caching.
• APIs. An OpenAI-compatible API server, plus Anthropic Messages API and gRPC support.
• Hardware. NVIDIA GPUs, AMD GPUs and x86, ARM and PowerPC CPUs, with hardware plugins extending it to other accelerators including Google TPUs and Apple silicon.

The shared detail: OpenAI-compatible endpoints

All three can expose a model through an API shaped like OpenAI's (Ollama alongside its own REST API). In practice that means code written against a hosted API can often be pointed at a local server by changing the base URL and model name - which is why these tools appear in agent and coding setups as well as in chat.

Related

• Model Context Protocol explained - how AI applications connect to tools, whether the model is local or hosted.
• The open source behind this site.
• MCP servers list.