What is the best runtime to run an LLM on-device inside a mobile or desktop app?

As of October 28, 2026: llama.cpp is the default for cross-platform apps (MIT, GGUF models, new builds daily); Google's LiteRT-LM (Apache 2.0, v0.18.1) is the best choice for Android with GPU and NPU acceleration and also runs on iOS, web and desktop; ExecuTorch (BSD, v1.5.1) is best when your model is in PyTorch and you need vendor NPU backends; MLX LM (MIT, v0.32.2) is fastest to adopt on Apple silicon Macs; ONNX Runtime GenAI (MIT, v0.17.0) is the Windows path through DirectML, QNN and Foundry Local.

Which on-device runtime supports NPUs?

LiteRT-LM accelerates on GPU and NPU and powers on-device AI in Chrome, Chromebook Plus and Pixel Watch. ExecuTorch delegates graph regions to CPU, GPU, NPU or DSP backends and falls back to portable CPU kernels. ONNX Runtime GenAI supports the Qualcomm QNN execution provider alongside CUDA, DirectML, OpenVINO and WebGPU. llama.cpp is mainly CPU and GPU.

What size model can run on a phone?

Roughly 1–4B parameters at 4-bit quantization for interactive use on current flagship phones. For example, Liquid AI's LFM3-1.7B, released October 27, 2026, needs about 1 GB of memory at 4-bit. Larger models fit on laptops with 16 GB or more of unified memory.

Should I use Ollama inside my app?

Ollama is a desktop server, not an embeddable mobile library. It suits desktop tools that can ask users to install it or that bundle it as a background service. For an app you ship through an app store, embed llama.cpp, LiteRT-LM, ExecuTorch or MLC LLM directly.

Quick Answer

Best On-Device LLM Runtimes for Mobile and Desktop Apps 2026

Published: October 28, 2026

The short answer

Pick the runtime by the platform you ship on first, then by the model format you already have. The best on-device LLM runtimes, as of October 28, 2026 (versions and licences from each project’s GitHub page):

Runtime	Licence / version	Platforms	Acceleration	Model format	Best for
llama.cpp	MIT; daily builds (b11946, Oct 28)	macOS, Linux, Windows, iOS, Android	CPU, Metal, CUDA, Vulkan and more	GGUF	Cross-platform apps, widest model support
LiteRT-LM (Google)	Apache 2.0; v0.18.1 (Oct 20)	Android, iOS, web, desktop, IoT	GPU, NPU	`.litertlm`	Android apps; Gemma models
ExecuTorch (PyTorch)	BSD; v1.5.1 (Sep 23)	iOS, Android, embedded, desktop	CPU, GPU, NPU, DSP via backend delegates	`.pte`	Teams with PyTorch models and vendor NPUs
MLX LM (Apple)	MIT; v0.32.2 (Oct 16)	Apple silicon Macs	Apple GPU via MLX	MLX safetensors	Mac apps and local fine-tuning
ONNX Runtime GenAI (Microsoft)	MIT; v0.17.0 (Sep 28)	Windows, Linux, Mac, Android, iOS	CUDA, DirectML, QNN, OpenVINO, WebGPU	ONNX	Windows apps, Copilot+ PCs
MLC LLM	Apache 2.0	iOS, Android, web, desktop	Metal, Vulkan, CUDA, WebGPU	Compiled MLC	WebGPU in the browser

Runtime notes

llama.cpp is the safe default: about 132,600 GitHub stars, new builds several times a day and day-one support for most open-weight releases in GGUF. Ollama and LM Studio build on it, so a model that runs in those desktop apps usually runs in your own llama.cpp integration. The trade-off is limited NPU use.

LiteRT-LM is Google’s production layer for LLMs on LiteRT, the runtime behind on-device generative features in Chrome, Chromebook Plus and Pixel Watch. Version 0.18 added EmbeddingGemma 2 multimodal embeddings, an OpenAI-compatible /v1/embeddings endpoint in litert-lm serve and on-demand KV-cache growth on NPU. It has Kotlin, Swift, JavaScript, Python and C++ APIs, function calling, and vision and audio inputs.

ExecuTorch exports a PyTorch model once with torch.export and produces a backend-specific .pte file per target, delegating supported graph regions to the device’s accelerators. Selective builds keep only the operators you need, which keeps app size down. Its versioning contract guarantees a .pte built on stable APIs loads on at least the next non-patch runtime release.

MLX LM generates text and fine-tunes models on Apple silicon, including LoRA and full fine-tuning on quantized models, with thousands of Hugging Face models one command away. It is a Mac tool: for iPhone apps use llama.cpp, LiteRT-LM, ExecuTorch or Apple’s own Foundation Models framework.

ONNX Runtime GenAI handles tokenization, sampling, KV cache and constrained decoding for tool calling on top of ONNX Runtime, and powers Foundry Local, Windows ML and the VS Code AI Toolkit. It is the most direct route to Qualcomm NPUs on Windows on Arm.

MLC LLM compiles models for each target and is the most mature option for running a model in the browser on WebGPU.

How to choose

One codebase for iOS, Android and desktop: llama.cpp with a GGUF model.
Android-first, battery matters: LiteRT-LM on GPU or NPU.
Custom PyTorch model, specific phone chips: ExecuTorch with the vendor backend.
Mac-only app or local fine-tuning: MLX LM.
Windows desktop app: ONNX Runtime GenAI or Foundry Local.
Runs in a web page: MLC LLM (WebLLM) or LiteRT-LM’s web build.

Then pick a model that fits the memory budget. New small models land often: Liquid AI’s LFM3 family ships GGUF, MLX and ONNX builds, and the 1.7B text model needs about 1 GB at 4-bit. For model rankings see best on-device AI models; for Apple’s own stack see Apple Core AI vs Foundation Models vs MLX.

What to test before you ship

Time to first token and tokens per second on your lowest supported device, not your development machine.
Peak memory, including the KV cache at your longest context; the OS will kill an app that exceeds it.
App size after bundling the runtime and the model, or a download-on-first-run flow.
Thermal throttling over a 10-minute session.
Licences of both runtime and model — runtimes here are permissive, but model licences vary.

Last verified: October 28, 2026.

The short answer

Runtime notes

How to choose

What to test before you ship

Sources