llama.cpp

The C++ inference engine that powers almost every local LLM tool.

Quick Facts

Attribute	Value
Pricing	Free
License	MIT
Best For	Maximum performance, customization
Language	C/C++
Format	GGUF
Founded	2023

What is llama.cpp?

llama.cpp is the foundational C++ library for running LLMs on consumer hardware. It pioneered efficient inference through quantization, making models that once required enterprise GPUs run on laptops.

Most local LLM tools—Ollama, LM Studio, Jan, GPT4All—use llama.cpp under the hood. Using llama.cpp directly gives you maximum control and performance at the cost of a more complex setup.

Key Features

• GGUF Format - Efficient model storage and loading
• Quantization - 4-bit, 8-bit, and mixed precision
• GPU Offloading - NVIDIA, AMD, Apple Silicon
• CPU Optimized - AVX, AVX2, AVX-512 support
• Server Mode - OpenAI-compatible API server
• Batch Processing - Efficient multi-request handling
• Speculative Decoding - Faster token generation
• Vision Support - Multimodal model support

Installation

# Clone repository
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# Build (CPU)
make

# Build with CUDA (NVIDIA)
make GGML_CUDA=1

# Build with Metal (Apple Silicon)
make GGML_METAL=1

# Run a model
./llama-cli -m model.gguf -p "Hello"

Quantization Levels

Type	Size vs F16	Quality	Use Case
Q2_K	30%	Lowest	Extreme constraints
Q4_K_M	45%	Good	Balanced
Q5_K_M	55%	Better	Quality focus
Q8_0	55%	Excellent	If RAM allows
F16	100%	Perfect	Full quality

Pros & Cons

Pros:

• Maximum performance possible
• Full control over inference
• Foundation for other tools
• Active development
• Cutting-edge features first

Cons:

• Requires compilation
• No GUI
• Steeper learning curve
• Manual model management

Alternatives

• Ollama - User-friendly wrapper
• LM Studio - GUI wrapper
• vLLM - High-throughput serving

FAQ

Should I use llama.cpp directly? Only if you need maximum performance or specific features. Ollama/LM Studio are easier for most users.

What’s GGUF? GGUF (GPT-Generated Unified Format) is the model file format for llama.cpp. It replaced GGML.

Can I convert models to GGUF? Yes, llama.cpp includes conversion scripts for PyTorch/Transformers models.

Is Apple Silicon good for local LLMs? Excellent. M1/M2/M3/M4 chips have unified memory and optimized Metal support in llama.cpp.

---

Last verified: 2026-03-04