TL;DR

Magnitude is an Apache-2.0 local inference server that does the one job Ollama and LM Studio deliberately leave to you: deciding what to run. It profiles your machine, measures actual memory bandwidth, ranks complete model configurations for that hardware, then downloads, tunes, and serves the one you pick — and writes your coding agent’s config file so the agent switches over to it.

  • Repo: magnitudedev/magnitude — 3,370 stars, 243 forks, TypeScript, Apache 2.0, created June 12, 2026.
  • Momentum: roughly 1,400 stars in the last week, which is what put it on GitHub Trending.
  • Maturity: young. CLI is at v0.0.11 (published September 2, 2026), with 0.0.12-alpha.0 already out. Treat it as alpha.
  • The pitch in one line: it is not a model runner, it is a model chooser that also runs the model.
  • Harnesses supported: Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, and Cline — plus an optional built-in harness.
  • Platforms: macOS and Linux natively; Windows through WSL.

Quick Reference

ItemDetail
Repogithub.com/magnitudedev/magnitude
LicenseApache License 2.0
LanguageTypeScript (CLI) + Rust inference engine over llama.cpp
Installnpm i -g @magnitudedev/cli
Package@magnitudedev/cli (v0.0.11 latest stable)
Service port127.0.0.1:10100 (loopback only)
APIsOpenAI-compatible + Anthropic-compatible
Model formatGGUF
Docsdocs.magnitude.dev
Data dir~/.magnitude/

The Problem Magnitude Actually Solves

Getting a model to run locally has been solved for years. Ollama does it, LM Studio does it, llama.cpp does it. What none of them do is tell you which model to run on your machine.

That gap matters more for agents than for chat, and the reason is structural. A chat session is a few thousand tokens. An agent trajectory accumulates for twenty or thirty turns, and every turn drags along the whole conversation history, every command run, every tool call and result. That context sits in RAM alongside the weights, and on long runs it routinely grows larger than the weights themselves.

Three things follow, and they’re why “just use Ollama” is weaker advice for agents than for chat:

  1. Quantization damage is not forgiving. A slightly degraded answer is fine in conversation. There is no partial credit on a tool call — it’s either structurally valid JSON or the loop breaks.
  2. Slow speeds compound instead of resetting. 20 tok/s feels fine when you’re reading along as it types. Across a twenty-step agent loop, it’s unusable.
  3. The agent runs flat out for minutes, which is exactly when memory pressure and thermal throttling appear.

The search space has four interacting axes: which model, how heavily it’s compressed, how the runtime is tuned, and how much context you configure. The honest industry practice today is to download twenty gigabytes and find out.

Why you can’t just read a spec sheet

Producing one token requires reading the model’s weights out of memory. So the ceiling on generation speed is roughly memory bandwidth ÷ size of the weights being read. Compute barely enters into it. That’s why a high TFLOPs rating doesn’t rescue a slow generation rate, and why configurations copied from a forum post so often disappoint — they were tuned against someone else’s memory bus.

Two examples show how sharp the edges are:

  • Capacity cliffs. Qwen3.6 35B-A3B is an MoE model — 35B total parameters, ~3B active per token — but all experts must be resident when routing picks, so the full 35B sits in memory regardless. At 8-bit that’s ~38GB (fine on 64GB, impossible in BF16 at ~70GB). A 16GB machine never loads it at any setting: even 4-bit is ~20GB before a single token of context.
  • Equal bit depth, unequal damage. Gemma 4 E2B ships as a quantization-aware-training model and holds most of its accuracy at 4-bit. Liquid LFM2.5 2.6B, at the same width, was compressed after training and gives up more. Bit depth tells you file size. It does not tell you what you gave up.

Magnitude’s bet is that this decision belongs to the layer that can see the hardware, not to whoever happens to be installing the tool.

How It Works

1. It profiles the machine

Magnitude detects processor, memory, architecture, and acceleration (Metal or CUDA). Then it does the part that separates measurement from arithmetic: it measures achieved memory bandwidth and runs short test inferences to see how the machine behaves in practice — before downloading anything large. Two machines with identical spec sheets can differ on real throughput because of architecture, thermal behavior, and whatever else is competing for memory. The profile is cached and recalibrated when needed.

2. It ranks complete configurations

You don’t get a list of models — you get complete configurations, each naming a model, a quantization, a context size, and an expected speed range. Magnitude ranks up to ten of them using model intelligence, estimated generation speed, quantization quality, configured context size, and physical memory.

A Fast ↔ Smart slider sits across the top, and moving it re-runs the recommendation against your hardware rather than just re-sorting a list. Speed is reported honestly as a range like ~36–48 tok/s, not a single flattering number.

3. It runs the model like an agent server, not a chat server

The inference engine is written in Rust on top of llama.cpp, installed and managed by Magnitude, so there’s no separate runtime to configure. The agent-specific behavior is where it earns its keep:

  • On-demand loading. Models stay on disk until requested, then unload when idle or when the machine needs memory. Only the first request after an unload pays load time.
  • Memory protection. It re-checks free memory before every load and stops a running model if memory gets dangerously low — before inference destabilizes the machine.
  • Context is preserved, not silently shrunk. Leftover capacity goes to concurrency rather than quietly cutting each request’s context. This is a bigger deal than it sounds (see below).
  • Prefill reuse. In ongoing sessions it reuses compatible prompt state and processes only new input.
  • Format normalization. Local models differ in reasoning formats, tool-call formats, chat templates, and history conventions. Magnitude normalizes them so a harness can swap models without model-specific code.

The concurrency trap, and why it’s the best argument for this tool

Concurrency trades directly against how much context each request keeps. Set it too high on a constrained machine and your context window silently shrinks. Nothing crashes. The logs look fine. The agent just gets quietly worse at long tasks — forgetting what it did ten steps ago — and the natural conclusion is “this model isn’t smart enough.”

That failure mode is invisible, and it’s almost certainly responsible for a chunk of the “local models can’t do agentic work” consensus.

Getting Started

The manual path is two commands:

npm i -g @magnitudedev/cli
magnitude setup

Interactive setup profiles the hardware, shows the recommendations, downloads your pick, and connects it to your harness.

The agent-first path is the interesting one — you paste a prompt into the agent you already use:

Set up local models for me with the Magnitude CLI. Install it with
`npm i -g @magnitudedev/cli` (or my package manager), then run
`magnitude docs onboarding` and follow the instructions.

Your agent profiles the hardware, walks you through the options, downloads what you choose, and rewrites its own config to point at the local model. Every non-interactive command supports --json, which is what makes this workable for agents rather than a gimmick.

The CLI worth knowing

magnitude service status              # installation, runtime, active model
magnitude catalog list                # what fits this machine, with state
magnitude catalog pull <model-id>     # install or update a catalog model
magnitude models status               # installed models + residency
magnitude models load <model-id>      # load by canonical ID
magnitude models stop                 # stop the active model
magnitude connections list            # supported + installed harnesses
magnitude connections add claude-code --set-model <model-id>
magnitude connections sync            # refresh harness config
magnitude docs onboarding             # the agent-facing setup workflow

Talking to it directly

The background service listens on loopback at http://127.0.0.1:10100 and speaks two dialects:

APIBase URL
OpenAI-compatiblehttp://127.0.0.1:10100/inference/v1
Anthropic-compatiblehttp://127.0.0.1:10100/inference/anthropic

So anything that speaks either protocol works without adapters:

curl http://127.0.0.1:10100/inference/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma-4-e2b",
    "messages": [{"role": "user", "content": "Refactor this function."}]
  }'

The OpenAI side supports model listing, Chat Completions, and Responses; the Anthropic side supports Messages and token counting. That Anthropic-compatible endpoint is the quiet reason Claude Code drops in cleanly.

Where things live

~/.magnitude/models/                   # managed local models
~/.magnitude/cache/                    # metadata, hardware profiles
~/.magnitude/config.json               # settings and model selections
~/.magnitude/harness-connections.json  # managed harness connections

Deleting ~/.magnitude/cache rebuilds derived data without losing downloaded models. The docs explicitly warn against nuking all of ~/.magnitude as a reflex troubleshooting step.

A Real Run

The most detailed independent walkthrough so far comes from Daily Dose of DS, who ran it on an Apple M5 with 10 cores and 16GB of unified memory. The profile finished in under a minute.

The balanced recommendation came back as Gemma 4 E2B at 4-bit QAT — a 5B dense text-and-vision model holding 50K context in 4.6GB, predicted at 43–51 tok/s, accuracy rated very high. Below it sat Liquid LFM2.5 2.6B and Qwen3.5 4B at various quantization levels.

Two details there show the tool isn’t just marketing itself:

  1. Predicted speed is a range, and its bottom sits near the point where an agent loop stops feeling usable. Magnitude doesn’t round that up.
  2. The balanced pick had no speculative decoding available, while the model beneath it did. That’s a real trade, not a bug — Gemma 4 E2B is faster and more accurate at the same bit depth, and gives up the drafter to get there.

They connected it to the Pi harness, turned off wifi, and had the agent review internal client files for PII. It completed the task offline and correctly, with nothing leaving the machine — the product claim, tested the only way it can honestly be tested.

On speculative decoding

Since reading the weights is the expensive part, the trick is getting more out of each read: a small fast model proposes several tokens ahead, and the real model verifies the whole proposal in one pass. Correct guesses yield several tokens for the cost of one. Whether it pays off depends on the model pairing and your bandwidth — exactly the judgment that needs a hardware profile, which is why Magnitude sets it rather than exposing it as a flag.

Honest Limitations

This is a young project and it shows. Being specific:

  • It’s alpha. v0.0.11 stable, 59 npm versions published, a 0.0.12-alpha.0 already out, and commits landing daily. Expect breaking changes.
  • Small community, real bus factor. 3,370 stars but only ~21 watchers and 17 open issues on a repo created in June 2026. Most contributions come from a handful of people around founder Tom Greenwald.
  • GGUF and llama.cpp only. No vLLM, no native MLX, no ExLlama. On Apple silicon an MLX-native path can beat llama.cpp on some models — you give that up for the automation.
  • Loopback only. The service binds 127.0.0.1, with no documented LAN-serving story. That rules out the homelab “one GPU box, many clients” pattern without your own proxy.
  • Not a production serving layer. Single-machine, single-user, on-demand loading. For multi-user throughput you still want vLLM.
  • The catalog is curated. Outside-catalog GGUF models work only via the Hugging Face CLI into the HF cache, followed by a restart to refresh inventory.
  • Windows is WSL-only.
  • No published benchmarks. No reproducible suite of its own, so the speed estimates — refreshingly presented as ranges — are the tool grading its own homework.
  • Physics still applies. A well-chosen 5B model is still a 5B model. Magnitude removes configuration error, not the capability gap against frontier models.

How It Compares

ToolChooses the model for youAgent-tuned servingWrites harness configBest for
Magnitude✅ hardware-profiled ranking✅ on-demand, memory-guarded✅Agents on local models
Ollama❌ you pick a tag⚠️ general-purpose❌Simplest general local LLM
LM Studio⚠️ compatibility hints⚠️ chat-oriented❌GUI exploration
llama.cpp❌❌ manual flags❌Maximum control
vLLM❌❌ throughput-oriented❌Multi-user GPU serving

Magnitude isn’t competing with llama.cpp — it runs llama.cpp. It competes with your afternoon, and with the failure mode where someone tries a local model once, picks a configuration nobody measured for them, and concludes their hardware isn’t good enough.

Who Should Use This

Use it if: you want a coding agent running fully offline; compliance, client-data, or air-gap constraints make hosted inference a non-starter; you have decent hardware and don’t know what it can run; or you burn tokens on high-volume mechanical work where per-token pricing dominates.

Skip it if: you need frontier-model quality on hard problems; you’re serving multiple users; you’re on bare Windows; you’ve already tuned llama.cpp yourself; or you can’t tolerate alpha tooling in your critical path.

FAQ

Is Magnitude free and open source?

Yes — Apache License 2.0, all of it, including the inference engine. No token costs, API keys, or rate limits. Your only cost is hardware and electricity.

How is this different from just using Ollama?

Ollama runs the model you name. Magnitude decides which model, quant, and context size to run by profiling your machine, tunes speculative decoding and concurrency for it, and writes your agent’s config. As the project’s FAQ puts it: an agent setting up Ollama is guessing — it doesn’t know your hardware, which quant fits, or how fast it’ll go.

Does any data leave my machine?

No. Prompts, files, and models stay local. Once Magnitude and a model are downloaded it runs fully offline — verified in the Daily Dose of DS test with wifi off.

Which coding agents does it work with?

Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, and Cline, plus an optional built-in harness. Because it exposes OpenAI- and Anthropic-compatible endpoints on 127.0.0.1:10100, most other tools speaking either protocol can be pointed at it manually.

What hardware do I need?

No fixed minimum — Magnitude profiles what you have and recommends accordingly. A 16GB Apple M5 got a usable 5B recommendation at 43–51 tok/s. Memory bandwidth, not core count, sets your token rate.

Can I use models outside the catalog?

Yes. Download compatible GGUF packages into your Hugging Face Hub cache with the HF CLI, then restart Magnitude to refresh inventory.

Do I have to manage it after setup?

No. It runs headless, loads models when the agent needs them, and unloads them when idle or memory tightens. Your agent can install or switch models via the CLI anytime.

Is it production-ready?

Not as a multi-user serving layer, and v0.0.11 says the rest. For a single developer running an agent on their own machine it’s usable today — expect frequent updates.

Verdict

Magnitude treats model selection as the product rather than an exercise left to the reader, and it targets the exact seam where local inference keeps failing agent workloads: not “can this run?” but “is this configured for the machine it’s running on?”

The measurement-over-arithmetic approach — profiling real bandwidth, running test inferences before downloading 20GB, reporting speed as an honest range, refusing to silently trade context for concurrency — is the right design. The Anthropic-compatible endpoint and --json on every command show it was built for agents to drive, not just for humans to click through.

It’s also 0.0.11 software with a three-month-old repo and a small maintainer pool, bound to llama.cpp and loopback. If you want offline agent coding and have hardware you don’t understand, install it this week. If you need stable infrastructure, watch it for a few months — but watch it, because this is the layer the local-agent stack has been missing.

Sources